IP-over-SCSI

2001/12/17

A "final" release was upload. Now all the things works well and we get near 23MB/s with direct connection, and near 14MB/s using TCP/IP. Not bad for a SCSI system. We don't have time to explain everything but now the token stays less time in each computer when there is data to send.

The direct connection make a perfect companion to an adapted MPI library and we hope that someone starts this project soon. For more information see the SPMDlib initiative at http://w3.ualg.pt/~dubuf/spmd.html.

Now for the graph. You can see a comparision between FastEthernet, TCP/IP using SCSI and a direct connection using the SCSI bus.

As you can not see, the direct connection is allways faster than ethernet. Not bad. TCP/IP over SCSI with low data sizes is slower than ethernet. Not very good, but useful. With diferent controlers, without token, it would be faster. Using FTP we got 13.45MB/s.

There are also the tests programs we used to get that graphic up there.

Have Fun... :)))

2001/10/29

A new release of the files is ready to go. We call it the "Almond Release". It works really well. Now we have all the speed we can take from the SCSI bus. This is the maximum we can get. Basicaly, there is 3 operating modes for SCSI:

8 bits, 5 MHZ (Maximum 5MB/s)
16 bits, 5 MHZ (Maximum 10MB/s)
16 bits, what the controler can take. With a SYM53C875 (SCSI-II) we got 40MB/s.

If we need some more speed we can get some newer board (like a SYM53C895) with Ultra3 SCSI that can get to the 160MB/s... Not bad. With this kind of speed, the problem is no longer the SCSI BUS, but the PCI BUS. One of this boards can use all the PCI bus, and it will not work at maximum speed...

What we did? Whell, we basicaly turned on WIDE, so that we can get a 16 bits bus, with 2 times the speed of a normal SCSI. Then we turned on Syncronous communication. With this 2 things we get a 20MHz clock in the bus and 2 bytes per cycle, so we got 40MB/s bettewn the computer. Not bad... One 100Mb/s ethernet board can get to the 11MB/s mark, so we got something near 4 times faster... :)))

Now the problem is not the speed of the bus, but the way it works. Because of the Token we implemented to control the SCSI bus, we can not get this kind of speed with TCP. With UDP we got half the maximum speed of the controler. So, using FTP, with a 40MB/s bus we got 5.86MB/s. Using UDP, we got 18.39MB/s. A realy big diference.

The next step is to put this thing working better. We are thinking how to change the token so that we got something better with TCP and UDP. If we take the token out and use only UDP, we can reach something near 40MB/s. But we need the token...

We will try to change the way we comunicate with the TCP/IP stack. First, when the computer dosen't have the token, it will stop the TCP/IP stack. This can bring a lot of delays. So we will change it so that the TCP/IP stack is allways working, but we got a kernel list that saves the packets, and send them when there is a token. Then we will check the new speed, and if it is better, we will stay with it...

Another thing we will do is a char device that will send packets from one board to another. You may ask why. It's easy. TCP/IP uses 14% of the bandwith with protocol. Without it, it will be faster talking bettwen computers. And we don't need the token. We will only use sequential communications, so the token is not necessary.

We are open to sugestions in how to improve the token. The problem is in the linux kernel timers. They got a resolution of 10ms. This is the minimum time that we can use to mark the token time. To use something less that 10ms, we can not use timers. One of the soluctions we got was to use the Scheduler queue to get a function working. The only problem is that it uses a lot of CPU (near 30-45% in a SMP system). So... How can we use a better resolution without occuping a lot of processor power? We don't know... If you can help, send us a mail.

2001/10/02

After a long time studing for final exams and deserved vacations, we finaly got some time to make this thing work... And it works... Now we have a SCSI link bettewen 2 computers using TCP/IP with any protocol (we didn't tested HTTP) with a fenonomal speed of 3.0MB/s ( or for the network freaks over there, 24Mb/s) without any optimizations, using only 8 bits of the SCSI bus (no WIDE or SYNC negotiation).

If we turn on WIDE/SINC, maybe we can get 2 or 4 times better results. As this board is only 16 bits, we can't use 32 bits to transfer data. With 16 bits we maybe get something like 6MB/s(48Mb/s).

The solutions for our problem was taken from the token-ring network. We use a software token to say witch board will send packets. Now that token is using a 10 ms interval for each board. The next idea is to implement a queue in the driver so that we can check how many packets we can send, and if the queue is empty, will send the token imediatly.

When we implement this ideia, will tell how it works here...

Why do we need the token. It's easy. The SCSI bus is half duplex. We can only send data from one side to the other at a time. Now when the controler wants to send a packet it must do some things first. One of them is the SELECT. SELECT says we want to use the bus. When the board do a SELECT there are 3 things that can happen.

SELECT works. We have the bus
There is another board doing the SELECT and we lose arbitation
Something else

The first case is the good one. The third there is no problem. The second is our problem. If the 2 controlers where trying to do a SELECT at the same time, the one with higher ID will win the arbitation, but it whant's to comunicate with the board that is waiting in the select... Big problem. With the token, only one of them can SELECT at a given time, so our problem is no more... :))

2001/04/22

Finaly we found our problem with the TCP/IP stack. It all started by one more invetigation of the structure of the TCP/IP stack. We found a function (tcp_v4_rcv) that receives and checks a packet. So we put a lot of printk's to see where in this function it would fail and reject the packet. The results was that this kernel said that he rejected the packet because it was not destined to it. Basicly he though that the packet was to another host in the network. This may happen when we are snifing all packets of a network or it is a multicast packet not direct to us (I don't know any more situations or even if these are correct).

So we saw where this variable as changed (skb->pkt_type). It was changed when we see the type of a packet, in the ethernet arp functions (eth_type_trans in linux/net/eth/eth.c). But we know that the packet is to our host, so we simply said that in the packet, and it works... We can now do traceroute, FTP,Telnet,HTTP,... Anything that uses the network, we can use it.

So we found another problem... This problem is not unexpected. It relates (we think) to the way that SCSI system work. There is a a piority to who can use the SCSI bus. The order is:

7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8

So the board with SCSI ID 7 can use the bus even if another host whants it also. So if we are doing a FTP from ID 7 to ID 6 (as is our case), the computer sends a series of packets to the other side, then waits for the ACK. But the ID 6 can not respond while 7 is talking. So the ACK's does not came... The a big problem happens. The SCSI system begins to timeout, so it trys to reset the SCSI bus. When the other board senses the reset, it also resets the bus... Big problem...

Now we are thinking of ways to remove this problems.

Fortunaly all the base of the system is working, and we can try anything. Tomorow we will try to use UDP to send a lot of fast packet to the other side. As the UDP do not reply with anything, only one host will be working. The problem of priority will not appear. If the problem is related to this, then it will work. If it is not related to ID priority, it will not work...

2001/05/16

The snh module is gone. We didn't needed it, so i just remove it. Now the sending and receiving is done in sn.c. By doing this i also removed a tasklet that was working in there. So the code in p5 is obsolete. I don't include it anymore in the distribution... If anyone whant to try to change it to current API, be my guest... Just send me a email and I will give it to you.

The problem continues...

2001/05/11 (first status report)

Done:

Target mode in the SYM53C8XX implemented and working right (SYM53C8XX)
High level SCSI communication module (SN)
Helper module to receive data from the scsi low level driver to the upper level done (SNH)
Test module to send data on the the scsi bus using a char device (Need some adjustments to work with latest sn) (P5)
TCP/IP stack to SN interface module working (SCSINET)

Know Problems

Wide and syncronous negotiation on the scsi driver not working
Anything trough SCSINET but ping dosen't work

Comentary

After long motnhs working in this project we can now say:

PING-Over-SCSI is working!!!!! WHEEEEEEEEEEEEEEEE!!!!!!!!!

The problem now is that anything we send that involves TCP or UDP does not work. The packet is sent to the stack, but nothing comes from the stack. By using ipgrab we got several trace from the TCP/IP stack to see what is happening. To understand this network packets dumps, see here for a description of the system.

Origin	Destination	Program used	Dump in origin	Dump in destination	status
ping	pong	ping	here	here	works
pong	ping	ping	here	here	works
ping	pong	traceroute	here	here	works
pong	ping	traceroute	here	here	not working
ping	pong	telnet	here	here	not working
pong	ping	telnet	here	here	not working

If you read the dumps, you can see that in one side of the network, the packets arrive allright, are sent to the TCP/IP stack, but nothing is sent from the stack. The strangest thing is that traceroute is working one way but not the other. The only diference bettewn the two computers is that on SCSI ID is 7 (PING), the other (PONG) is 6.

If we change the SCSI ID (PONG=7,PING=6) the traceroute works the other way around.

Architeture

The ideia is to make the smallest of modules with the minimum of functionality. So we decided to do several modules that is very simple. The most complicated part of the project was to make the SCSI board work as a target. As you can see in the next figure, the SCSI board is the lowest of all the levels.

In our case Machine 1 is PING and Machine 2 is PONG.

Note that all of this can change before we end the project.

Modules

SYM53C8XX

This was based in the normal Linux SCSI driver.

In SCSI there is a Iniciator and several Targets. Normaly the initiators are the SCSI boards inside the computer and the target all that is in the SCSI bus.

To make our system work we need to make the SCSI Board work as a Initiator and a Target at the same time. Several boards are available, but we choose to work with Symbios version 53C875 as they are cheap, and available everywhere.

The main work was to implement the target mode, so that we could receive data from the other side. We begun with stunding SCRIPTS (the internal language in the sym53C8xx chip), and then implement all that was needed for the target to work.

In this moment, we use the SCSI command INQUIRY to check if the other computer is there, then we can send data with the SCSI command WRITE10.

If you notice, in the sending side (Machine 1) the data passes trough scsi_mod, but in the receiving data it goes directly to SN. This happen because SCSI_MOD is prepared to receive data if and only if it is the initiator. But in a SCSI network, the board work as target, so SCSI_MOD is not prepared to receive data.

SN

This objective of this module is to get data from anything above or bellow and send it to any scsi target. It will not check the data, the size, nothing at all. It will only send it to a target. Also it identifys any target that it can work with. This is necessary so that the SCSI subsystem in the Linux kernel works.

P5

This module is usable again. We needed it to use with MPI.

This module intention was to test the other modules, by giving a interface with user space using the computer filesystem. We created some devices and everything that is written to them will be sent to the other side of the scsi bus. That way we could test all of SN,SNH and SYM53C8XX.

At this time, this module is completly broken. There was some changes in SN that take care of it. We don't need this, it was only a test module (and a way to learn something more of the linux kernel), so we don't change it anymore. It has a lot of bugs.

SCSINET

This is a interface to the TCP/IP stack inside the linux kernel. It works like a Ethernet, with ethernet headers. With this we got a new network interface (sn0) that can be used just like a normal ethernet. This moment (2001/05/11) we are working in this module, discovering and squashing bugs.

Source Code

The source is available here in tar.gz format. To compile you will need a 2.4 kernel. To use you need two computers with symbios 53c875 boards connect by a HDB68<->HDB68 cable externely.

This intructions are a little bad... Send me a email if you whant to compile this thing. To compile:

put the code in sym in /usr/linux/drivers/scsi
link sn/sn.h to /usr/linux/drivers/scsi
Configure your kernel with support for the sym53c8xx board
compile the modules
link /usr/src/linux/drivers/scsi/{scsi.h,hosts.h,scsi_obsolete.h} to sn
Check the Makefile of sn (sn/Makefile)
run make in sn

To use:

Check that the SCSI boards are connected
insert scsi_mod if isn't compiled in the kernel
insert sn.o
insert sym53c8xx.o
in the other computer insert all of the above. It should detect the first computer. If not, something wrong.
in the first computer, write:
echo "scsi add-single-device 0 0 7 0" > /proc/scsi/scsi
changing the 7 to the SCSI ID of the second computer. Now doing "cat /proc/scsi/scsi" should show a device with the ID of the other computer.
insert scnet.o
cat /proc/net/dev and check if there is a sn0 device there.
now you can use you SCSI boards as a network device like ethernet. (ifconfig,ping,traceroute,ftp,telnet,...)

Contact info

If you are in Faro, Portugal, you can check us in Vision Lab of Universidade do Algarve (Campus de Gambelas).

If not, you can mail us to:

Pedro Semeano

Luis Sismeiro

IP-Over-SCSI Project Page

Index

Introduction

Current Status