Erwan proposes an existing XMPP server called OpenFire in combination with Arduino's that send JSON messages and a server app calle TAK-server that handles different message streams etc.
He's writing the system in Java, more info here.
XMPP (jabber protocol) - @ okno server there's already a simple system in place that can read in scripts and sends them over xmpp. the program can take any script running on the same comp thru stdin and relayes it into the xmpp server.
PACHUBE; FORMAT: XML or JSON; RULE: max 50 requests every 3 minutes (eg. 1 request every 3.6 seconds); ALSO: triggers (when conditions user sets have been satisfied, eg. value change, value exceeds threshold, value equals, etc.); ALSO: historical values from the last 24 hours; EXAMPLE: connected plant http://www.pachube.com/feeds/2263
noiseart's protocol - written in java; each node sent self defined messages to a central server which posted them to anyone who asked for them. a message might look like this: 'wind speed 4'. a remote location asks for all 'wind' messages. there was 1 server, and then many remote installation pieces connected to it. people used java, flash, MAX, and PD to connect
(1) Internet protocols in general are not intended to be used for real-time data transfer. Even at the lowest level, these protocols are designed to move data from point A to point B going through several intermediate nodes on the way. They have to balance your data with all the other data flowing through this elaborate (massive) system.
(2) Data is sent in packets. Packets have to travel through the operating system (Windows, Mac OS X, Linux, or some code running on an embedded processor such as an Arduino) then they are passed off to a network interface (a WiFi card or Ethernet). Every piece of software and hardware that handles your packet may alter it in order to meet certain hardware constraints or to make the overall data flow more efficient from the network's point of view. Typically, operating systems and network interfaces will collect data until they have a minimum amount of data to send. If you are sending 2 byte messages then the network will try to collect several of these messages into 1 packet before actually transmitting anything.
(3) There are ways to control how your data is turned into packets, and how those packets are sent and received. (a) Most network drivers and interfaces have a built-in timeout of 200 ms. If no new data is received in this time, then any existing data is sent. So if you are sending small pieces of data from a Wind Clock, and you send a new message every 150 ms, then your messages will be added together before being sent. But if you send messages every 250 ms then your messages should be sent one at a time. Of course, this timeout is implemented differently by different operating systems and different hardware manufacturers. (b) Network drivers and interfaces use buffers (a place to store data before it is sent), and the size of a buffer is usually a multiple of 256 bytes. For real-time transmission you should always use packet sizes that are equal divisions or multiples of 256 (i.e. 32 bytes, 64, 128, 256, 512). (If you send odd sized, or variable sized packets, then the hardware will attempt to make them multiples of 256 for you, and you end up fighting the hardware).
(4) As a result, for real-time messages, it makes sense to use a message protocol that packages data into objects of a specific size. (For example: I wrote a networked video player .. a program that plays many channels of video in sync .. and it forwards packets of 64 bytes quite nicely at 33 ms +/- 3 ms intervals to 8 different Mac OS X machines).
(1) There are two techniques that are generally used for sending messages to a server. (a) Most script based systems will open a connection to the server, send a message, close the connection to the server, and then end the script. The next time you run the script, it opens a new connection to the server, etc. This is a good method for cases where you are sending data once every few minutes or hours. But the overhead of constantly opening and closing the network connection will start to overwhelm the system if you are sending messages faster then every 20 to 30 seconds. (b) The other technique is to open a “persistent” connection to the server. Your Arduino can talk to a low level Internet interface, or send data to a laptop that runs some “bridge” software. But in any case, you open a single connection to the server and then keep it open for as long as you are sending data.
(2) Persistent connections have issues too, especially if the communication is one way. i.e. the Wind Clock transmits data to the server but never receives anything. This is the case with mp3 audio servers (like icecast). They send the audio to the listener, but the listener never sends anything back to the server to say if it is really listening or not. As a result, audio servers will often server audio to non-existent (or missing) listeners, which isn't very efficient or useful. For most messaging applications, it is a good idea to have some kind of acknowledgment that the message was actual received and not lost in space somewhere. In the case of the “Scrambler” (the primitive message server we used in 2003 for connecting installation works), the server would keep track of the time between messages from the different installations, and if there was a gap of over 10 seconds, it would broadcast a timestamp to everyone just to make sure that all the network connections were still alive and well.
(1) It is fairly easy to develop a custom server if it only has to handle a small number of clients (30 to 50 connections). If we want to handle more connections then it might be best to use an existing message server. This would involve using something like XMPP (which is based in XML I believe). We could still “invent” a custom message protocol, and then we would “wrap” it in the whatever the XMPP server needs.
(2) I think we want to be able to support a variety of artist-made-devices that are sending and receiving data to/from a server. And I'm assuming that this is going to be a real-time system where messages are received and then forwarded immediately to anyone who wants them. (As opposed to a system that would store the messages in an archive and then provide access to them at a much later date). BUT I could be wrong about this .. storing messages can be useful too!
(3) We want the protocol to be simple enough to run on an Arduino. Ideally it should be able to run on an Arduino that is directly connected to the Internet. But with XMPP it may be necessary to run it through a larger system to translate it into XML and to deal with the message server login and handshaking, etc.
(1) It would be a good idea to start with the individual sources of data .. the artworks! And from there we can try to list the types of data that people want to send and receive. Once we have a list (even just a preliminary one), we can figure out how best to move the data around from the technical perspective.
(2) We can also think about a general approach to the message data – should it be human readable, or machine friendly? With the “Scrambler” we chose to make the messages human readable so that we could also output them as text to a screen and show people what was running through the system in a quick and fun way. But it also possible to make the messages easier to handle at the machine level, especially for programs running on the Arduino.