* Implemented the following changes.
* Remove NUM_NODES_PER_CHUNK
* Remove NUM_EDGES_PER_CHUNK
* Remove the dependency between no. of edge files per edge type and no. of partitions
* Remove the dependency between no. of edge feature files per edge type and no. of partitions
* Remove the dependency between no. of edge feature files and no. of edge files per edge type.
* Remove the dependency between no. of node feature files and no. of partitions
* Add “node_type_counts”. This will be a list of integers. Each integer will represent total count of a node-type. The index in this list and the index in the “node_type” will be the same for a given node-type.
* Add “edge_type_counts”. This will be a list of integers. Each integer will represent total count of an edge-type. The index in this list and the index in the “edge_type” list will be the same for a given edge-type.
* Applying lintrunner patch.
* Adding missing keys to the metadata in the unit test framework.
* lintrunner patch.
* Resolving CI test failures due to merge conflicts.
* Applying lintrunner patch
* applying lintrunner patch
* Replacing tabspace with spaces - to satisfy lintrunner
* Fixing the CI Test Failure cases.
* Applying lintrunner patch
* lintrunner complaining about a blank line.
* Resolving issues with print statement for NoneType
* Removed tests for the arbitrary chunks tests. Since this functionality is not supported anymore.
* Addressing CI review comments.
* addressing CI review comments
* lintrunner patch
* lintrunner patch.
* Addressing CI review comments.
* lintrunner patch.
* Flexible pipeline - Initial commit
1. Implementation of flexible pipeline feature.
2. With this implementation, the pipeline now supports multiple partitions per process. And also assumes that num_partitions is always a multiple of num_processes.
* Update test_dist_part.py
* Code changes to address review comments
* Code refactoring of exchange_features function into two functions for better readability
* Upadting test_dist_part to fix merge issues with the master branch
* corrected variable names...
* Fixed code refactoring issues.
* Provide missing function arguments to exchange_feature function
* Providing the missing function argument to fix error.
* Provide missing function argument to 'get_shuffle_nids' function.
* Repositioned a variable within its scope.
* Removed tab space which is causing the indentation problem
* Fix issue with the CI test framework, which is the root cause for the failure of the CI tests.
1. Now we read files specific to the partition-id and store this data separately, identified by the local_part_id, in the local process.
2. Similarly as above, we also differentiate the node and edge features type_ids with the same keys as above.
3. These above two changes will help up to get the appropriate feature data during the feature exchange and send to the destination process correctly.
* Correct the parametrization for the CI unit test cases.
* Addressing Rui's code review comments.
* Addressing code review comments.
* [Dist] enable to partition many chunks into less partitions via pipeline
* refine
* add meta file for num_parts, add more tests, refine docstring
* remove args.num_parts
* create pydantic class for partition metadata
* refine
* rename json file
* [DistPart] expose timeout config for process group
* refine code
* Update tools/distpartitioning/data_proc_pipeline.py
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
* Adding launch script and wrapper script to trigger distributed graph partitioning pipeline as defined in the UX document
1. dispatch_data.py is a wrapper script which builds the command and triggers the distributed partitioning pipeline
2. distgraphlaunch.py is the main python script which triggers the pipeline and to simplify its usage dispatch_data.py is included as a wrapper script around it.
* Added code to auto-detect python version and retrieve some parameters from the input metadata json file
1. Auto detect python version
2. Read the metadata json file and extract some parameters to pass to the user defined command which is used to trigger the pipeline.
* Updated the json file name to metadata.json file per UX documentation
1. Renamed json file name per UX documentation.
* address comments
* fix
* fix doc
* use unbuffered logging to cure anxiety
* cure more anxiety
* Update tools/dispatch_data.py
Co-authored-by: Minjie Wang <minjie.wang@nyu.edu>
* oops
Co-authored-by: Quan Gan <coin2028@hotmail.com>
Co-authored-by: Minjie Wang <minjie.wang@nyu.edu>
