Tuesday, February 7, 2012

You have to store a huge graph of 1billion of nodes


1 comment:

  1. OK, I'll bite, sounds like it could be a fun problem.

    But before I can even hope to suggest a solution I'd need a little more information. So here it goes:

    1. What are the requirements for the node data? Do we only need to persist it? Does it need to be traversed? Is the set of nodes immutable?

    2. Can you give me more information about each node? What kind of data does it contain? Is each node's data immutable? Any special requirements for the node data? Expected size requirement per node? Is the structure of the nodes homogeneous?

    3. More information about the graph and its topology? Is it directional? Is it acyclic? Is it a tree? Does it have connected component graphs? Any significant vertex centrality? Is it likely to contain any cliques?

    Maybe a little more information about its use and the agents that need to traverse it.

    4. Does an agent need to have have access the entire graph at once? Do we need to keep a representation of the whole graph in memory? Will there be more than one agent accessing the node data and graph structure at the same time? In case the graph data is mutable and accessible by multiple agents, does the data and structure need to be consistent across all agents at all times?

    Oh my that is a lot, better stop here before going completely off track :)