Sunday, May 8, 2016

building an interactive tree with d3.js php and sqlite: an annotated webliography

I want to make a tree with rectangular nodes containing text, such that I can add nodes or remove nodes on the fly. I want the tree to be "multiplayer": everybody who visits the site sees the same tree, and when one of them modifies it, it the updates are applied to everyone's view.
I know nothing about d3, and very little about PHP, but a bit about sqlite and Apache.
For this project I'm using a minimal number of libraries, and a very simple server setup, which leads to some inefficiencies. The server is Ubuntu 14.04 with Apache 2.4 with PHP 5.6. Client side the only library I'm using is d3.js 3.5.

A consequence of the simplicity is that I don't have a good way to push data to clients, so I can't use long polling or web sockets. Instead, I rely on repeated client requests for updates, which adds some latency, but hopefully not too much.


Requirements for graph:
Nodes are boxes that can contain about 50 words
terminal nodes can be added or removed (capability to add or remove internal nodes is not necessary)
root is on the left
subtrees can be expanded or collapsed
zoom-in zoom-out capability
pan capability
sort nodes with the older siblings above newer siblings


Let's get this figured out!


(see bottom of page for a link to a working version of the code)

Leanring d3.js:

Really concise explanation of data-joins. Read this!
http://bost.ocks.org/mike/join/

Documentation for the selections feature of d3. Read this too...

https://github.com/mbostock/d3/wiki/Selections

another good introduction to D3
http://code.hazzens.com/d3tut/lesson_0.html

Making an updatable table with d3
https://vis4.net/blog/posts/making-html-tables-in-d3-doesnt-need-to-be-a-pain/

Mouseovers:
http://stackoverflow.com/questions/23584748/how-to-display-a-text-when-mouseover-a-node-in-d3-force-layout

detecting keypresses:
http://stackoverflow.com/questions/15261447/how-do-i-capture-keystroke-events-in-d3-force-layout

scroll to bottom of div:
http://stackoverflow.com/questions/14246768/d3js-how-to-scroll-or-tween-properties

Example and explanation of dragging and zooming:
https://bl.ocks.org/mbostock/6123708

super-simple zoom and pan:
https://coderwall.com/p/psogia/simplest-way-to-add-zoom-pan-on-d3-js

For my application is was critical that I tied the zoom event handler to the <svg> element, but had the first inner <g> block take the transformation. If I didn't do this, then everything was jumpy and extreme and bad.

D3 as an AJAX library:

put scripts at the end of the page to execute them after the rest of the page has loaded.
http://stackoverflow.com/questions/9899372/pure-javascript-equivalent-to-jquerys-ready-how-to-call-a-function-when-the

d3 can act as a replacement for jquery for ajax requests
blog.webkid.io/replacing-jquery-with-d3/

to append arbitrary xml content to a d3 selection use d3.select().node().appendChild()
an example of an ajax request is:
d3.html("login.html", function(error, html) {
    d3.select("#main").node().appendChild(html);
});
http://stackoverflow.com/questions/21209549/embed-and-refer-to-an-external-svg-via-d3-and-or-javascript

Generic javascript AJAX:

Use the setTimeout() function to set a function to get called after a delay. For example, after receiving an update from the server, set a timeout to request another update after a certain number of seconds.
https://developer.mozilla.org/en-US/Add-ons/Code_snippets/Timers

javascript:

remove specific items from a javascript array with splice.
http://stackoverflow.com/questions/5767325/remove-a-particular-element-from-an-array-in-javascript

SVG:

We want to make the node a box around some text. To do this, we need to know the size of text.
One way to do this is with the svg getComputedTextLength() function. (this will not give us the height of the text).
Another way is to use the offsetHeight or offsetWidth functions.

wrapping text:
https://bl.ocks.org/mbostock/7555321

Generating and modifying the graph:

Official documentation of the D3 Tree Layout.
Create a tree with d3.layout.tree()
get nodes with tree(root)
get links with tree.links(nodes)

Essentially what tree does is take as input a hierarchical JSON object, and return a similar JSON object, but with x and y coordinates added. Passing that new object to links then returns a list of (source, target) pairs, where source and target are

set sorting withtree.sort([comparator]) #this will be useful for making
https://github.com/mbostock/d3/wiki/Tree-Layout

Here's a stackoverflow question with exactly the same issue I have. The code here isn't terribly helpful, but the link in the comments is (see the next link down on this page)
http://stackoverflow.com/questions/25942826/d3-tree-add-and-remove-nodes

Tutorial about how to make a modifiable tree
http://www.d3noob.org/2014/01/tree-diagrams-in-d3js_11.html
the book associated with the previous link
https://leanpub.com/D3-Tips-and-Tricks/read

Tutorial on how to update a d3 svg based on changing json data
http://pothibo.com/2013/09/d3-js-how-to-handle-dynamic-json-data/

d3 trees, by default are top-down trees. To make them left-to-right trees, just give x coordinates instead of y coordinates, and vice versa. For example:
    enter_node.append("svg:circle")
      .attr("r", 10)
      .attr("cx", function(d) { return d.y; })
      .attr("cy", function(d) { return d.x; });
and
   .enter().append("path")
    .attr("class", "link")
    .attr("d", d3.svg.diagonal().projection(function(d) {return[d.y, d.x]}));

A better way is to write a recursive function go through and swap x and y coordinates immediately after generating the nodes object. For example:
nodes = tree.nodes(root);
nodes.map(function(n) {var t = n.x; n.x = n.y; n.y = t});
links = tree.links(nodes);

http://stackoverflow.com/questions/18099430/how-to-change-orientation-of-a-d3-tree-layout-by-90-degrees


Use "Tree.nodeSize" to change node size (apparently only works in d3 v3+):
http://codepen.io/augbog/pen/LEXZKK
http://stackoverflow.com/questions/32839957/tree-nodesize-not-working-in-d3-tree-graph-to-inc-the-space-between-nodes

By default, the edges look pretty bad (concave). To make them convex, swap the x and y (for reasons I don't fully understand yet).
http://stackoverflow.com/questions/15007877/how-to-use-the-d3-diagonal-function-to-draw-curved-lines


PHP and SQLite3:
For queries that don't return results, use "exec"
http://www.php.net/manual/en/sqlite3.exec.php

For queries that do return results, use "query"
http://us3.php.net/manual/en/sqlite3.query.php

Better yet, use the "prepare" function with execute
http://us3.php.net/manual/en/sqlite3.prepare.php


To process a SELECT COUNT operation, use: "querySingle"
http://stackoverflow.com/questions/2586598/using-sqlite3-in-php-how-to-count-the-number-of-rows-in-a-result-set

To send stuff to an error log use the command: error_log("Error message\n", 3, "path_to/php.log");

http://stackoverflow.com/questions/15530039/how-to-write-to-error-log-file-in-php

use "list" to separate multiple return values
http://php.net/manual/en/functions.returning-values.php


Associative arrays and numeric indexed arrays are the same thing in PHP:
http://php.net/manual/en/language.types.array.php

To convert an array to json, use json_encode:
http://php.net/manual/en/function.json-encode.php 

array map (perform function on all array elements), see top comment for how to pass keys and values:
http://php.net/manual/en/function.array-map.php

CSS:

You can access html data elements from css selectors using brackets:

.node_rect[data-status='not_approved']
https://developer.mozilla.org/en-US/docs/Web/Guide/HTML/Using_data_attributes

Examples:
Lots of examples of trees
http://christopheviau.com/d3list/gallery.html#visualizationType=tree

Stack overflow answer with zooming and panning of a tree
http://stackoverflow.com/questions/17405638/d3-js-zooming-and-panning-a-collapsible-tree-diagram

collapsible tree from example from Bostock
http://bl.ocks.org/mbostock/4339083 


Special lessons and considerations:

One frustrating aspect of D3 is that the tree function (and I presume other layout functions) modifies the data it takes as an input. I presume this is for performance reasons (because javascript copies objects by refrence, and d3 doesn't want to make deep copies of anything, so it adds references to the new object, then modifies them), but the behavior is a bit frustrating and surprising. I think the solution is to format the node data as soon as it is received from the server so that it always has a consistent layout.

When you call a function on a selection, the function is called individually on all members of that selection. So when I select all nodes, and each node has only one rect as a child, I should use nodes.select('rect'), and not nodes.selectAll('rect')

If you feed an empty object to layout.tree it does not return an empty object. What it returns is a list with one member: an object with properties depth=0, x=0, and y=0. That is: a tree with 1 node. I think this is another quirky consequence of JavaScript objects. JavaScript objects are never truly empty because they have lots of stuff they inherit just by being an object. In other words: it is impossible to use tree.nodes to create an empty tree.

The solution to this is, after creating the link nodes (there won't be any, because there's only one node). Clear out the nodes list (nodes=[]). Then downstream code using .data(nodes) will work, and you can safely update the tree so that it contains no nodes. The first 6 lines of code of my graph_refresh function are thus:

function graph_refresh(root) {
 var nodes = tree.nodes(root);
 nodes.map(function(n) {var t = n.x; n.x = n.y; n.y = t});
 var links = tree.links(nodes);
 if (!(nodes[0].hasOwnProperty('id'))) {
  nodes=[];
 }


Result:

here's the link

here are the instructions:

There are currently three users:
Sean
user
User2
The passwords for all of them are "123"
You can make new users too if you want.


Once you make and join a chart, you can modify it by entering commands into the chat window.
There are currently 5 possible commands:
reply:  @X message (where X is the number of an existing node. To make the first node, use @0. The other member has to approve of a node before anyone can reply to it)
approve: #X (where X is the number of an unapproved node)
remove: !rmX (X must be an unapproved node that you are the author of)
modify: !mdX (X must be an unapproved node that you are the author of)
make public: !p (once both users have entered this, then any registered user can look at the graph)
Any time you enter something that doesn't start with !,@, or #, it will be interpreted as chat, and go into the chat window.
You can use the mousewheel to zoom in and out, and click and hold to pan around.


 

No comments:

Post a Comment