IP-Over-SCSI Project Page
This project is in sourceforge. See our project page at sourceforge
The direct connection make a perfect companion to an adapted MPI library and we hope that someone starts this project soon. For more information see the SPMDlib initiative at http://w3.ualg.pt/~dubuf/spmd.html.
Now for the graph. You can see a comparision between FastEthernet, TCP/IP using SCSI and a direct connection using the SCSI bus.
As you can not see, the direct connection is allways faster than ethernet. Not bad. TCP/IP over SCSI with low data sizes is slower than ethernet. Not very good, but useful. With diferent controlers, without token, it would be faster. Using FTP we got 13.45MB/s.
There are also the tests programs we used to get that graphic up there.
Have Fun... :)))
What we did? Whell, we basicaly turned on WIDE, so that we can get a 16 bits bus, with 2 times the speed of a normal SCSI. Then we turned on Syncronous communication. With this 2 things we get a 20MHz clock in the bus and 2 bytes per cycle, so we got 40MB/s bettewn the computer. Not bad... One 100Mb/s ethernet board can get to the 11MB/s mark, so we got something near 4 times faster... :)))
Now the problem is not the speed of the bus, but the way it works. Because of the Token we implemented to control the SCSI bus, we can not get this kind of speed with TCP. With UDP we got half the maximum speed of the controler. So, using FTP, with a 40MB/s bus we got 5.86MB/s. Using UDP, we got 18.39MB/s. A realy big diference.
The next step is to put this thing working better. We are thinking how to change the token so that we got something better with TCP and UDP. If we take the token out and use only UDP, we can reach something near 40MB/s. But we need the token...
We will try to change the way we comunicate with the TCP/IP stack. First, when the computer dosen't have the token, it will stop the TCP/IP stack. This can bring a lot of delays. So we will change it so that the TCP/IP stack is allways working, but we got a kernel list that saves the packets, and send them when there is a token. Then we will check the new speed, and if it is better, we will stay with it...
Another thing we will do is a char device that will send packets from one board to another. You may ask why. It's easy. TCP/IP uses 14% of the bandwith with protocol. Without it, it will be faster talking bettwen computers. And we don't need the token. We will only use sequential communications, so the token is not necessary.
We are open to sugestions in how to improve the token. The problem is in the linux kernel timers. They got a resolution of 10ms. This is the minimum time that we can use to mark the token time. To use something less that 10ms, we can not use timers. One of the soluctions we got was to use the Scheduler queue to get a function working. The only problem is that it uses a lot of CPU (near 30-45% in a SMP system). So... How can we use a better resolution without occuping a lot of processor power? We don't know... If you can help, send us a mail.
If we turn on WIDE/SINC, maybe we can get 2 or 4 times better results. As this board is only 16 bits, we can't use 32 bits to transfer data. With 16 bits we maybe get something like 6MB/s(48Mb/s).
The solutions for our problem was taken from the token-ring network. We use a software token to say witch board will send packets. Now that token is using a 10 ms interval for each board. The next idea is to implement a queue in the driver so that we can check how many packets we can send, and if the queue is empty, will send the token imediatly.
When we implement this ideia, will tell how it works here...
Why do we need the token. It's easy. The SCSI bus is half duplex. We can only send data from one side to the other at a time. Now when the controler wants to send a packet it must do some things first. One of them is the SELECT. SELECT says we want to use the bus. When the board do a SELECT there are 3 things that can happen.
So we saw where this variable as changed (skb->pkt_type). It was changed when we see the type of a packet, in the ethernet arp functions (eth_type_trans in linux/net/eth/eth.c). But we know that the packet is to our host, so we simply said that in the packet, and it works... We can now do traceroute, FTP,Telnet,HTTP,... Anything that uses the network, we can use it.
So we found another problem... This problem is not unexpected. It relates (we think) to the way that SCSI system work. There is a a piority to who can use the SCSI bus. The order is:
7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8
So the board with SCSI ID 7 can use the bus even if another host whants it also. So if we are doing a FTP from ID 7 to ID 6 (as is our case), the computer sends a series of packets to the other side, then waits for the ACK. But the ID 6 can not respond while 7 is talking. So the ACK's does not came... The a big problem happens. The SCSI system begins to timeout, so it trys to reset the SCSI bus. When the other board senses the reset, it also resets the bus... Big problem...
Now we are thinking of ways to remove this problems.
Fortunaly all the base of the system is working, and we can try anything. Tomorow we will try to use UDP to send a lot of fast packet to the other side. As the UDP do not reply with anything, only one host will be working. The problem of priority will not appear. If the problem is related to this, then it will work. If it is not related to ID priority, it will not work...
The problem continues...
PING-Over-SCSI is working!!!!! WHEEEEEEEEEEEEEEEE!!!!!!!!!
The problem now is that anything we send that involves TCP or UDP does not work. The packet is sent to the stack, but nothing comes from the stack. By using ipgrab we got several trace from the TCP/IP stack to see what is happening. To understand this network packets dumps, see here for a description of the system.
|Origin||Destination||Program used||Dump in origin||Dump in destination||status|
If you read the dumps, you can see that in one side of the network, the packets arrive allright, are sent to the TCP/IP stack, but nothing is sent from the stack. The strangest thing is that traceroute is working one way but not the other. The only diference bettewn the two computers is that on SCSI ID is 7 (PING), the other (PONG) is 6.
If we change the SCSI ID (PONG=7,PING=6) the traceroute works the other way around.
The ideia is to make the smallest of modules with the minimum of functionality. So we decided to do several modules
that is very simple. The most complicated part of the project was to make the SCSI board work as a target. As you can
see in the next figure, the SCSI board is the lowest of all the levels.
In our case Machine 1 is PING and Machine 2 is PONG.
Note that all of this can change before we end the project.
In SCSI there is a Iniciator and several Targets. Normaly the initiators are the SCSI boards inside the computer and the target all that is in the SCSI bus.
To make our system work we need to make the SCSI Board work as a Initiator and a Target at the same time. Several boards are available, but we choose to work with Symbios version 53C875 as they are cheap, and available everywhere.
The main work was to implement the target mode, so that we could receive data from the other side. We begun with stunding SCRIPTS (the internal language in the sym53C8xx chip), and then implement all that was needed for the target to work.
In this moment, we use the SCSI command INQUIRY to check if the other computer is there, then we can send data with the SCSI command WRITE10.
If you notice, in the sending side (Machine 1) the data passes trough scsi_mod, but in the receiving data it goes
directly to SN. This happen because SCSI_MOD is prepared to receive data if and only if it is the initiator. But in a
SCSI network, the board work as target, so SCSI_MOD is not prepared to receive data.
This objective of this module is to get data from anything above or bellow and send it to any scsi target. It will
not check the data, the size, nothing at all. It will only send it to a target. Also it identifys any target that
it can work with. This is necessary so that the SCSI subsystem in the Linux kernel works.
This module intention was to test the other modules, by giving a interface with user space using the computer filesystem. We created some devices and everything that is written to them will be sent to the other side of the scsi bus. That way we could test all of SN,SNH and SYM53C8XX.
At this time, this module is completly broken. There was some changes in SN that take care of it. We don't need this,
it was only a test module (and a way to learn something more of the linux kernel), so we don't change it anymore. It has
a lot of bugs.
This is a interface to the TCP/IP stack inside the linux kernel. It works like a Ethernet, with ethernet headers. With this
we got a new network interface (sn0) that can be used just like a normal ethernet. This moment (2001/05/11) we are working
in this module, discovering and squashing bugs.
The source is available here in tar.gz format. To compile you will
need a 2.4 kernel. To use you need two computers with symbios 53c875 boards connect by a HDB68<->HDB68
This intructions are a little bad... Send me a email if you whant to compile this thing. To compile:
echo "scsi add-single-device 0 0 7 0" > /proc/scsi/scsi
changing the 7 to the SCSI ID of the second computer. Now doing "cat /proc/scsi/scsi" should show a device with the ID of the other computer.
If not, you can mail us to: