Commit Graph

56 Commits

Author SHA1 Message Date
Muhammed Fatih BALIN
d1161229e0 [GraphBolt][CUDA] Cooperative Minibatching [1] - UniqueAndCompact. (#7765) 2024-09-03 17:45:03 -04:00
Muhammed Fatih BALIN
09ea3196e6 [GraphBolt][CUDA] Update CCCL to get backport fix. (#7716) 2024-08-17 00:42:19 -04:00
Muhammed Fatih BALIN
0d68130f92 [CUDA] Remove unused gpu_cache files and update. (#7699) 2024-08-14 20:44:16 -04:00
Muhammed Fatih BALIN
cb4604aca2 [Performance] Remove phmap dependency. (#7658) 2024-08-05 11:29:40 -04:00
Muhammed Fatih BALIN
04940ae766 [GraphBolt][CUDA] Update CCCL to 2.6.0. (#7636) 2024-08-02 19:43:13 -04:00
Muhammed Fatih BALIN
f724ec0a00 [GraphBolt] Change hash table for performance. (#7631) 2024-08-02 06:46:50 -04:00
Muhammed Fatih BALIN
105f0c6834 [GraphBolt] Parallel backend experimentation (#7596) 2024-08-01 06:37:25 -04:00
Muhammed Fatih BALIN
52651379a0 [GraphBolt][io_uring] Refactor and enable tests (#7506) 2024-07-09 04:06:09 -04:00
Muhammed Fatih BALIN
25c29537b9 [GraphBolt] Partitioned S3-FIFO Feature Cache implementation (#7492) 2024-07-04 22:24:15 -04:00
Muhammed Fatih BALIN
489671cd43 [Misc] Make code compilable with GCC 13 (#7363)
Co-authored-by: Hongzhi (Steve), Chen <chenhongzhi.nkcs@gmail.com>
2024-06-27 14:17:07 +08:00
Muhammed Fatih BALIN
ad5485be23 [GraphBolt][CUDA] Update cccl and cuco dependencies. (#7472) 2024-06-26 12:07:24 +08:00
Muhammed Fatih BALIN
67a897fa56 [CUDA] Update CCCL to 2.4.0. (#7375) 2024-05-06 13:48:07 +08:00
Muhammed Fatih BALIN
78df81015a [GraphBolt] Add optimized unique_and_compact_batched. (#7239) 2024-04-07 04:55:08 -04:00
Hongzhi (Steve), Chen
912164a883 [Misc] Update liburing version to avoid unexpected dirty commit after build. (#7275)
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
2024-04-07 14:05:25 +08:00
xiangyuzhi
0c3a4428c1 [Graphbolt] Add liburing dependency (#7149)
Co-authored-by: Ubuntu <ubuntu@ip-172-31-40-250.ap-northeast-1.compute.internal>
2024-02-29 14:48:55 +08:00
Muhammed Fatih BALIN
73e01d6daa [CUDA] Update CCCL to 2.3.0 (#7171) 2024-02-29 00:37:00 -05:00
Muhammed Fatih BALIN
99fcdfece7 [Bug] Update thrust to the new monorepo CCCL (#6166)
Co-authored-by: Rhett Ying <85214957+Rhett-Ying@users.noreply.github.com>
2023-12-05 15:22:36 +08:00
Xin Yao
726836976d [Misc] Upgrade googletest to v1.14.0 (#6273)
Signed-off-by: Xin Yao <xiny@nvidia.com>
2023-09-04 15:50:46 +08:00
Muhammed Fatih BALIN
69a532c1ab [Feature] Gpu cache for node and edge data (#4341)
Co-authored-by: xiny <xiny@nvidia.com>
2023-07-24 13:17:10 +08:00
Muhammed Fatih BALIN
83115794c2 [Performance][CUDA] Faster CSRToCOO (#5648)
Co-authored-by: Hongzhi (Steve), Chen <chenhongzhi.nkcs@gmail.com>
2023-07-14 12:57:38 +08:00
Rhett Ying
66c04855da [DistDGL] remove tensorpipe from third_party and no build (#5876) 2023-06-15 17:31:05 +08:00
Hongzhi (Steve), Chen
9ff56d2098 [Cleanup] Remove featgraph and unused TVM dependency. (#5767)
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
2023-06-02 15:14:31 +08:00
Hongzhi (Steve), Chen
a93e5578d6 [Misc] Remove xbyak (#5752)
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
2023-05-30 16:40:20 +08:00
Andrzej Kotłowski
6862e37259 Upgrade libxsmm (#5725)
Co-authored-by: Rhett Ying <85214957+Rhett-Ying@users.noreply.github.com>
2023-05-24 11:25:57 +08:00
Xin Yao
8d5d8962ad [Refactor] Replace third_party/nccl with PyTorch's NCCL backend (#4989)
* expose GeneratePermutation

* add sparse_all_to_all_push

* add sparse_all_to_all_pull

* add unit test

* handle world_size=1

* remove python nccl wrapper

* remove the nccl dependency

* use pinned memory to speedup D2H copy

* fix lint

* resolve comments

* fix lint

* fix ut

* resolve comments
2023-03-08 12:59:10 +08:00
Muhammed Fatih BALIN
bf264d00fe [Feature] (La)yer-Neigh(bor) sampling implementation (#4668)
* adding LABOR sampling

* add ladies and pladies samplers

* fix compile error after rebase

* add reference for ladies sampler

* Improve ladies implementation.

* weighted labor sampling initial implementation draft
fix indentation and small bug in ladies script

* importance_sampling currently doesn't work with weights

* fix weighted importance sampling

* move labor example into its own folder

* lint fixes

* Improve documentation

* remove examples from the main PR

* fix linting by not using c++17 features

* fix documentation of labor_sampler.py

* update documentation for labor.py

* reformat the labor.py file with black

* fix linting errors

* replace exception use with if

* fix typo in error comment

* fixing win64 build for ci

* fixing weighted implementation, works now.

* fix bug in the weighted case and importance_sampling==0

* address part of the reviews

* remove unused code paths from cuda

* remove unused code path from cpu side

* remove extra features of labor making use of random seed.

* fix exclude_edges bug

* remove pcg and seed logic from cpu implementation, seed logic should still work for cuda.

* minor style change

* refactor CPU implementation, take out the importance_sampling probability computation into a function.

* improve CUDAWorkspaceAllocator

* refactor importance_sampling part out to a function

* minor optimization

* fix linting issue

* Revert "remove pcg and seed logic from cpu implementation, seed logic should still work for cuda."

This reverts commit c250e07ac6d7e13f57e79e8a2c2f098d777378c2.

* Revert "remove extra features of labor making use of random seed."

This reverts commit 7f99034353080308f4783f27d9a08bea343fb796.

* fix the documentation

* disable NIDs

* improve the documentation in the code

* use the stream argument in pcg32 instead of skipping ahead t times, can discard the use of hashmap now since it is faster this way.

* fix linting issue

* address another round of reviews

* further optimize CPU LABOR sampling implementation

* fix linting error

* update the comment

* reformat

* rename and rephrase comment

* fix formatting according to new linting specs

* fix compile error due to renaming, fix linting.

* lint

* rename DGLHeteroGraph to DGLGraph to match master

* replace other occurrences of DGLHeteroGraph to DGLGraph

Co-authored-by: Muhammed Fatih BALIN <m.f.balin@gmail.com>
Co-authored-by: Kaan Sancak <kaansnck@gmail.com>
Co-authored-by: Quan Gan <coin2028@hotmail.com>
2022-11-22 09:03:02 +08:00
Xin Yao
cded5b80fe [Feature] Bump DLPack to v0.7 and decouple DLPack from the core library (#4454)
* rename `DLContext` to `DGLContext`

* rename `kDLGPU` to `kDLCUDA`

* replace DLTensor with DGLArray

* fix linting

* Unify DGLType and DLDataType to DGLDataType

* Fix FFI

* rename DLDeviceType to DGLDeviceType

* decouple dlpack from the core library

* fix bug

* fix lint

* fix merge

* fix build

* address comments

* rename dl_converter to dlpack_convert

* remove redundant comments
2022-09-19 16:02:43 +08:00
Israt Nisa
06608f847c [Cleanup] Remove duplicate entries of CUB submodule (issue# 4395) (#4499)
* remove third_part/cub

* remove from third_party

Co-authored-by: Israt Nisa <nisisrat@amazon.com>
Co-authored-by: Xin Yao <xiny@nvidia.com>
2022-09-05 13:23:08 +08:00
Xin Yao
86c81b4e92 [Feature] Add CUDA Weighted Neighborhood Sampling (#4064)
* add weighted sampling without replacement (A-Chao)

* improve Algorithm A-Chao with block-wise prefix sum

* correctly fill out_idxs

* implement weighted sampling with replacement

* small fix

* merge host-side code of weighted/uniform sampling

* enable unit tests for cuda weighted sampling

* move thrust/cub wrapper to the cmake file

* update docs accordingly

* fix linting

* fix linting

* fix unit test

* Bump external CUB/Thrust versions

* Fix code style and update description of algorithm design

* [Feature] GPU support weighted graph neighbor sampling
commit by pengqirong(OPPO)

* merge pengqirong's implementation

* revert the change to cub and thrust

* fix linting

* use DeviceSegmentedSort for better performance

* add more comments

* add necessary notes

* add necessary notes

* resolve some comments

* define THRUST_CUB_WRAPPED_NAMESPACE

* fix doc

Co-authored-by: 彭齐荣 <657017034@qq.com>
2022-07-29 11:08:48 +08:00
nv-dlasalle
bf649d940c [Bugfix][Performance] Update nccl to 2.11.4 (#3761)
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
2022-02-23 13:51:41 +08:00
Jinjing Zhou
a3ce780d29 [RPC] Use tensorpipe for rpc communication (#3335)
* doesn't know whether works

* add change

* fix

* fix

* fix

* remove

* revert

* lint

* lint

* fix

* revert

* lint

* fix

* only build rpc on linux

* lint

* lint

* fix build on windows

* fix windows

* remove old test

* fix cmake

* Revert "remove old test"

This reverts commit f1ea75c777.

* fix windows

* fix

* fix

* fix indent

* fix indent

* address comment

* fix

* fix

* fix

* fix

* fix

* lint

* fix indent

* fix lint

* add introduction

* fix

* lint

* lint

* add more logs

* fix

* update xbyak for C++14 with gcc5

* Remove channels

* fix

* add test script

* fix

* remove unused file

* fix lint

* add timeout
2021-12-06 20:03:33 +08:00
Jinjing Zhou
769718dfcf Bring back thrust for backward compatibility (#3562) 2021-12-03 16:30:35 +08:00
Jinjing Zhou
bba88cd6dc remove unused dependency (#3504) 2021-11-15 14:21:58 -08:00
sanchit-misra
fac75e166b [CPU][Kernel] Single socket spmm (#3024)
* optimizations of spmm for CPU

* Added names of contributors

* Minor code cleanup

* Moved the spmm optimization code to a new header file

* Moved to DGL's logging method

* removed duplicate code between SpMMSumCsr and SpMMCmpCsr

* Changes made to follow Google coding style

* Fixed lint errors in spmm.h

* Fixed some lint errors from spmm_blocking_libxsmm.h

* Fixed lint errors from spmm_blocking_libxsmm.h

* Added comments to SpMMCreateLibxsmmKernel

* to enable building of tests, and other cosmetic changes

* disabling libxsmm on windows

* Put a condition to avoid opt impl for FP64 as libxsmm does not have FP64 support yet

* cosmetic changes and documentation

* cosmetic changes

* to pass lint tests

* replaced multiple allocations for buffers of indices and edges with a single allocation

Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
2021-07-13 11:18:29 +08:00
nv-dlasalle
66eb240d15 [Bugfix] Include NCCL as a submodule (#2934)
* Add NCCL as a submodule

* Allow using third_party/nccl or system nccl

* Add nccl_external as a dependency

* Fix conditional

Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
2021-05-25 09:46:16 +08:00
Tianqi Zhang (张天启)
e83d0a8007 [Feature] Add kd-tree implementation (CPU) for kNN (#2767)
* add submodule nanoflann

* finish python API for knn

* finish ndarray adaptor

* finish cpu-kdtree version of knn

* use openmp

* add endline

* upt

* upt

* fix format and code style

* upt

* add warning for gpu-cpu copy

* avoid contiguous copy

Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
Co-authored-by: Tong He <hetong007@gmail.com>
2021-04-09 17:39:34 +08:00
Zhi Lin
4208ce2b9e [Feature] Tvm integration (#2367)
Co-authored-by: Zihao Ye <expye@outlook.com>
2020-12-31 17:40:25 +08:00
pawelpiotrowicz
f8ebcd7f90 [Performance] Dynamic cpu kernel V3 for SpMMSumCsr all Ops (#2309)
* support AVX512

* env DGL_CPU_INTEL_KERNEL_ENABLED=1

* env DGL_CPU_INTEL_KERNEL_LOG=1

* Add unittest test_spmm.cc

Co-authored-by: Izabela Mazur <izabela.mazur@intel.com>
Co-authored-by: Michal Szarmach <michal.szarmach@intel.com>

Review patch
2020-11-17 15:12:23 +08:00
Minjie Wang
4fb0241bfb [CUDA] Add CUDA11 support (#2308)
* add support for cuda 11

* fix inc bug in pytorch 1.8

* poke ci

* fix

* small fix

* try fix

* try fix

Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com>
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
2020-11-07 22:16:51 +08:00
Jinjing Zhou
3234189b60 Fix hdfs error when writing big data (#2186)
* update dmlc-core to latest

* fix hdfs
2020-09-12 23:58:36 +08:00
Jinjing Zhou
628d9fc59d Fix build error with hdfs and update dmlc-core (#2107)
* update dmlc-core for hdfs build

* add hdfs support

* default off

* trigger ci
2020-08-26 15:10:21 +08:00
Da Zheng
729ff2ef38 Fix the performance issue of graph partitioning in new DGLGraph (#1934)
* fix perf.

* fix.

* accelerate metis.

* fix lint.

* use gklib.

* fix perf.

* fix.

* update metis.

* update launch script

* handle synchronized API.

* fix.

* fix example.

* fix dataloader.

* temp fix.

* temp fix omp.

* distinguish roles.

* initialize iterator of DistDataloader correctly.

* check the correctness of launch script.

* move feature copy to sampler.

* measure mem/network copy time.

* remove

* Revert "measure mem/network copy time."

This reverts commit 86cefdc14b.

* fix.

* fix

* fix.

* fix cmake.

* disable metis in windows.

* disable metis tests in windows.

* remove test for multigraph.

* fix test.

* fix.

* fix cmake.

* fix.

* revert.

Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-115.us-west-2.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal>
2020-08-09 18:33:05 -07:00
Da Zheng
0e896a921e update. (#1857)
Co-authored-by: Ubuntu <ubuntu@ip-172-31-30-164.us-west-2.compute.internal>
2020-07-24 10:47:57 -07:00
Quan (Andy) Gan
22a6ad6d9b [Windows] Compile METIS on Windows (#1771)
* make metis compilable on windows

* lint

Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com>
2020-07-09 15:43:50 +08:00
Minjie Wang
870da747ea [CUDA][Kernel] More CUDA kernels; Standardize the behavior for sorted COO/CSR (#1704)
* add cub; array cumsum

* CSRSliceRows

* fix warning

* operator << for ndarray; CSRSliceRows

* add CSRIsSorted

* add csr_sort

* inplace coosort and outplace csrsort

* WIP: coo is sorted

* mv cuda_utils

* add AllTrue utility

* csr sort

* coo sort

* coo2csr for sorted coo arrays

* CSRToCOO from sorted

* pass tests for the new kernel changes

* cannot use inplace sort

* lint

* try fix msvc error

* Fix g.copy_to and g.asnumbits; ToBlock no longer uses CSC

* stash

* revert some hack

* revert some changes

* address comments

* fix

* fix to_block unittest

* add todo note
2020-06-28 18:37:28 +08:00
IzabelaMazur
8a20b6c19c [Optimization] Replaced std::unordered_map by phmap::flat_hash_map (#1662)
Co-authored-by: Da Zheng <zhengda1936@gmail.com>
2020-06-21 10:27:29 -07:00
Da Zheng
0e153c4baf [Feature] add metis partitioning to DGL (#1308)
* add metis.

* add test.

* construct partition id.

* link to METIS github repo.

* update metis.

* add a tool for partitioning a graph.

* update metis.

* update.

* update.

* fix metis.

* fix lint

* fix indent.

* another way of building metis.

* disable metis in windows.

* test windows

* fix.

* disable metis for windows properly.

* fix for tensorflow.

* skip test for gpu.

* make graph symmetric

* address comments.

* more comments.

* fix compile

* fix a bug.

* add test.

* change the default #hops of HALO nodes.

Co-authored-by: Ubuntu <ubuntu@ip-172-31-26-167.us-east-2.compute.internal>
2020-03-07 23:24:03 -08:00
Chao Ma
cccde032f4 [kvstore] Performance improvement for distributed kvstore (#972)
* Performance improvment for distributed kvstore

* update

* update
2019-11-04 18:27:00 +08:00
Zihao Ye
9a0511c8e9 [NN] nn modules & examples update (#890)
* upd

* damn it

* fuck

* fuck pylint

* fudge

* remove some comments about MXNet

* upd

* upd

* damn it

* damn it

* fuck

* fuck

* upd

* upd

* pylint bastard

* upd

* upd

* upd

* upd

* upd

* upd

* upd

* upd

* upd
2019-11-03 17:29:42 -05:00
VoVAllen
0fb13f7b9d [Feature] Data format (#728)
* Add serialization

* add serialization

* add serialization

* lalalalalalalala

* lalalalalalalala

* serialize

* serialize

* nnn

* WIP: import tvm runtime node system

* WIP: object system

* containers

* tested basic container composition

* tested custom object

* tmp

* fix setattr bug

* tested object container return

* fix lint

* some comments about get/set state

* fix lint

* fix lint

* update cython

* fix cython

* ffi doc

* fix doc

* WIP: using object system for graph

* c++ side refactoring done; compiled

* remove stale apis

* fix bug in DGLGraphCreate; passed test_graph.py

* fix bug in python modify; passed utest for pytorch/cpu

* fix lint

* Add serialization

* Add serialization

* fix

* fix typo

* serialize with new ffi

* commit

* commit

* commit

* save

* save

* save

* save

* commit

* clean

* Delete tt2.py

* fix lint

* Add serialization

* fix lint 2

* fix lint

* fix lint

* fix lint

* fix lint

* Fix Lint

* Add serialization

* Change to Macro

* fix

* fix

* fix bugs

* refactor

* refactor

* updating dmlc-core to include force flag

* trying tempfile

* delete leaked pointer

* Fix assert

* fix assert

* add comment and test case

* add graph labels

* add load labels

* lint

* lint

* add graph labels

* lint

*  fix windows

* fix

* update dmlc-core to latest

* fix

* fix camel naming
2019-09-09 20:57:51 +08:00