...making Linux just a little more fun!

November 2008 (#156):


Mailbag

This month's answers created by:

[ Sayantini Ghosh, Amit Kumar Saha, Ben Okopnik, Dave Richardson, Faber Fedor, Kapil Hari Paranjape, Karl-Heinz Herrmann, René Pfeiffer, Neil Youngman, Paul Sephton, Rick Moen, Robos, Thomas Adam ]
...and you, our readers!

Gazette Matters


Undeliverable mail: Delivery reports about your e-mail

Rick Moen [rick at linuxmafia.com]


Tue, 30 Sep 2008 19:21:25 -0700

[[[ I've clipped out the offending message and tossed it into the /misc directory as http://linuxgazette.net/156/misc/lg/undeliverable if you actually are curious about what "these things" look like. -- Kat ]]]

Subscribers might (or might not) be curious about what I do with these things. Basically, Faber had it about right: I do indeed decide where to aim the Blacklist of Doom and fire freely.

In this case, IP address 207.65.4.4 (rex2.isdn.net), running proprietary MTA Communigate Pro, confessed that it had received (forged) mail from IP address 71.75.8.158, which at the time was falsely claiming to be "lists.linuxgazette.net" and conveying mail from tag@lists.linuxgazette.net to downstream recipient chet@multipro.com.

And that incompetently run MTA, on rex2.isdn.net, believed that particular barefaced lie -- and thus decided to send "back" to the forged, falsified sender address, tag@lists.linuxgazette.net, the scintillating intelligence that the illustrious Mr. Chet is over quota and sadly cannot accept "our" mail.

This is incompetent mail administration in part because rex2.isdn.net has no excuse for believing that lie: Merely conducting the most basic of checks of our SPF record would have revealed that 71.75.8.158 is not an authorised MX for lists.linuxgazette.net. Also, rex2 is ethically obliged to be particularly careful about what mail it accepts for redelivery to (e.g.) downstream hosts like multipro.com's MXes (mail exchangers), specifically because of the risk of "backscatter spam" of this sort.

In short, I've blackholed rex2.isdn.net completely on lists.linuxgazette.net (for all SMTP purposes), which is indeed hard-assed but ensures that it will not backscatter-spam us again. If there are any legitimate users on that mail host, they lose.

(As a reminder, TAG is particularly vulnerable to spam, because its posting model has to date prevented it from being set so that all mail from non-subscribed addresses is held for listadmin vetting. All other mailing lists I run duck that bullet, and are for all practical purposes spam-free.)


Still Searching


MPPP Problem

kuldeep singh [kuldeep_1985_ec at yahoo.co.in]


Sun, 5 Oct 2008 11:31:07 +0530 (IST)

[[[ I suspect this is "please do my homework", but as it wasn't responded to last month, I'm tossing it into the "Still Searching" pile. (Unsurprisingly, it was sent as html, adding to my annoyance.) -- Kat ]]]

hi

I am a student from IIT Delhi, India. I am working on a project in which i need to establish multilink PPP connection between two embedded ARM +board. The OS is linux(Debian).

Can u please tell me how to establish this mppp connection (configuring in Linux).

Please reply soon.

Thanks

Kuldeep

IIT Delhi


iptables configuration in debian

J.Bakshi [j.bakshi at icmail.net]


Wed, 8 Oct 2008 19:18:12 +0530

Hello friends,

I have a remote suse server where I have a shell script and in that I have the configuration of iptables firewall. so, after executing the script I get my firewall ready. I could also write a deamon there which can execute the script after booting the linux.

I wounder if there is anything in debian to do that. When checking iptables I have found the following

iptables
iptables-apply
iptables-multi
iptables-restore
iptables-save
iptables-xm

shall I just write a deamon which will then call the configuration file ? or there is already something in debian which I have missed ? Kindly suggest. Thanks


[OT] Booting mackbooks (was Help with Linux !)

Chris Bannister [mockingbird at earthlight.co.nz]


Wed, 29 Oct 2008 00:55:13 +1300

On Mon, Oct 27, 2008 at 06:38:09PM -0400, Faber J. Fedor wrote:

> On my Macbook, I boot off a CDROM by a) putting the CD into the drive,
> b) rebooting the machine, c) and hold down the 'c' key during the
> boot process.

Do they still eject the cd if you try and boot off them or just ignore it?

-- 
Chris.
======
I contend that we are both atheists. I just believe in one fewer god
than you do. When you understand why you dismiss all the other
possible gods, you will understand why I dismiss yours.
                                           -- Stephen F Roberts


Our Mailbag


deleted file recovery

J.Bakshi [j.bakshi at icmail.net]


Thu, 16 Oct 2008 20:24:54 +0530

Dear list,

I generally worked with Linux terminal. Frequently I have to do "rm" to delete files/folders which are not required any more. And this actually raised a oquestion in mind. what to do if some point of time I need a file/foldr which has been already deleted before few min. ago. I have googled a lot but have not found any open source tools which can recover the deleted ones. testdisk is there but it is totally different. there are some commercial GUI based tools available but I am looking for a CLI tool or technique which practically recover the deleted file/folder.

Hope some one can enlighten me. Thanks

[ Thread continues here (18 messages/31.21kB) ]


startx problem

J.Bakshi [j.bakshi at icmail.net]


Sun, 5 Oct 2008 22:58:03 +0530

Dear list,

I don't need kdm to run as a service because most of the time I access this linux box remotely. Only when I am at home then I need the desktop. That's why I have disabled the kdm. But after that "startx" gives an error

Module Loader present
Markers: (--) probed, (**) from config file, (==) default setting,
(++) from command line, (!!) notice, (II) informational,                                                                 
(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
                                                                                                                             
(==) Log file: "/var/log/Xorg.0.log", Time: Sun Oct  5 21:15:35 2008                                                                                                                          
(==) Using config file: "/etc/X11/xorg.conf"                                                                                                                         
(II) Module "i2c" already built-in                                                                                                                           
(II) Module "ddc" already built-in                                                                                                                         
(II) Module "ramdac" already built-in
                                                                                                                             
waiting for X server to shut down FreeFontPath: 
FPE "/usr/share/fonts/X11/misc"
                                                                                                                             
refcount is 2, should be 1; fixing.

Every time the same error from startx. If I start the kdm no problem at all !!

Kindly CC to me

[ Thread continues here (10 messages/19.31kB) ]


What is the policy of apt in lenny ?

J.Bakshi [j.bakshi at icmail.net]


Sun, 5 Oct 2008 22:56:49 +0530

Hello List,

I have tried to install the gdeskcal with its dependency "python python-central python-gtk2 python-numeric python-xml" in lenny and In return apt asking to install 170 packages with gnome gnome desktop and some other tools which really don't have any connection with gdeskcal !!! I did not see this in sarge. What is going on ?

I like to install gdeskcal with icewm. Why 170 pacakes ??

Thanks

[ Thread continues here (17 messages/18.20kB) ]


How to get fancy terminal ?

J.Bakshi [j.bakshi at icmail.net]


Wed, 8 Oct 2008 14:26:53 +0530

Hello list,

Once I tried Mandrake linux at my friend's PC. I have never seen that type of console in any other distro ( though I worked with redhat, debian, suse and puppy). The fonts are not so big which I see in debian or RH; they are neat and clean. more over at the bottom of the console there was a logo of mandrake !! Could any one suggest or give me any clue so that I can modify my present console to that fancy one ?

thanks

[ Thread continues here (2 messages/1.13kB) ]


Help with Linux !!!!!!!!!!!!!!!

bill jones [userwilber2003 at yahoo.com]


Mon, 27 Oct 2008 13:16:50 -0700 (PDT)

[[[ Mr. Jones sent this in as html mail, and also without much spacing in his punctuation. For ease of the readership, I've done some minor editing on his content. -- Kat ]]]

Hello.

I need some help and I'm hoping you can do it. Ok,here goes. I traded a guy at work for an Apple laptop with Linux installed. He got it from a guy who didn't have the Root password. How nice! I've tried to

+ reinstall Opensuse 10.2 to gain controll of the laptop,and it won't autorun. I made sure the discs were in ISO form. When the disc is in the cdrom

+ it shows how big the disc is but it doesn't show how much space is available. The memory shows that there's' 2.34mb avaliable. I'm very new at this

+ Linux and don't really understand it yet. So, do you think you'ins' can help me?

ON Opensuse site it said if it didn't start auto to open Bios setup and go from ther. I'm not sure how to do that ,or where to type in Bios. Any help would be greatly welcomed and appreaciated!

Thank-you,

-- 
Bill Jones
userwilber2003@yahoo.com

[ Thread continues here (4 messages/5.08kB) ]


apt sources.list for multimedia

J.Bakshi [j.bakshi at icmail.net]


Mon, 6 Oct 2008 23:09:25 +0530

Dear list,

I did a net installation of lenny. I have not found xmms and I think the source is mising from the configuration. Here is my sources.list

deb http://ftp.de.debian.org/debian lenny main contrib non-free
deb http://security.debian.org/ lenny/updates main contrib non-free
deb-src http://security.debian.org/ lenny/updates main contrin non-free

will it need additional source to have the multimedia ?

Thanks

[ Thread continues here (10 messages/17.23kB) ]


How to modify linux console ?

J.Bakshi [j.bakshi at icmail.net]


Fri, 10 Oct 2008 22:55:42 +0530

Dear all,

I am a debian user but once I got a chance to test Mandrake Linux and there I saw first time the magic of Linux console. First of all; the Mandrake console was neat and clean with high resolution support and Colourful text. More over a transparent Logo was there at the buttom of the screen. It was really nice looking and at the same time disclousing the fact what power is inbuilt inside Linux console!!

I compiled the kernel ( in my debian box ) with Fb and vesa support. Then I add the vga=791 at the kernel configuration line at grub.cfg and now I get high resolution screen with linux logo support at booting. But There are so many things still unresolved and I'll be grateful to the list if some one atlest show me the right direction

1> How can I test what max. resolution my monitor can suport with a command ?

2> High resolution can be opted by SVGA, VGA or fbdevice. What are the difference between these and how can I know which one is best for my Display card ?

3> How to get colorful text; like the text in knoppix during booting ?

4> and finally how can I get the tranparent logo at the butom of my console ?

A no. of question :-) may be these are already there in some document in the system itself. But currently I have no clue and it would be great if Someone atleast show me the direction following which I can be able to resolved all these myself.

Thanks

[ Thread continues here (9 messages/12.03kB) ]


(How do I run/install downloaded programs using linux?)

Hunt, Melissa [mhunt at Dewberry.com]


Wed, 29 Oct 2008 11:36:17 -0500

Im new to xandros linux...never used any linux operating system before...

How do I run/install downloaded programs using linux?

Wind0ws usually opts for an auto run or save. It seems linux wants to save. So once I have it saved (which I do) how do I actually run or install that downloaded program?

Thanks...

Melissa-Hope Hunt

-- 
Computer Aided Drafter
Dewberry
200 Broadacres Drive, 4th Floor
Bloomfield, NJ 07003-3154
973.338.9100 ext. 3116
973.338.5860 fax
www.dewberry.com <http://www.dewberry.com>

[ Thread continues here (7 messages/6.35kB) ]


ISP blues (was: iptables configuration in debain)

Rick Moen [rick at linuxmafia.com]


Thu, 9 Oct 2008 15:39:12 -0700

Hi, J.Bakshi. This note is to advise you that your choice of ISP is causing your mail to be interpreted upon receipt as suspect, when automatically parsed by spam-detection software. My SMTP software keeps kicking your mail addressed to this mailing list into an administrative queue, to verify that it's not spam. Even though I've whitelisted _you personally_, your ISP's IP address keeps failing several tests used within Spamassassin. Those are:

RCVD_IN_SORBS: This is fairly serious. Your ISP's SMTP server IP address (63.223.93.2) is in the SORBS (Spam and Open-Relay Blocking System) antispam blocklist, which means it's been extensively used for spamming activity (not by you).

RCVD_IN_SORBS_WEB: This is also fairly serious. That same IP is also in a separate SORBS blocklist of sites that are known to have exploitable vulnerability (e.g., FormMail scripts) that are widely abused by spammers to send mass-mail spam.

FORGED_RCVD_HELO: This refers to the fact that your ISP sends out a mistaken/malformed greeting string during its SMTP delivery attempts. In this case, IP address 63.223.93.2 (firewall.wahju.com) is sending a greeting string claiming it's "icsmail.net". Doing so violates mailing list technical standards and makes your ISP's SMTP host look like a spammer haven.

RCVD_IN_XBL: This is really serious. This means your ISP's SMTP server IP address is also in the Spamhaus XBL blocklist, described as "wholly incorporat[ing] data from two highly-trusted DNSBL sources, with tweaks by Spamhaus to maximise the data efficiency and lower False Positives. The main components are the CBL (Composite Block List) from cbl.abuseat.org, and the NJABL Open Proxy IPs list from www.njabl.org". Essentially, this means your ISP's mail server has been verified to be repeatedly used by spammers to a high degree of certainty.

I cannot really advise you, except to say that, unless/until some of the above facts about your ISP -- not your fault and not under your personal control -- change radically, you are likely to have significant problems with software classifying your mail as probable spam. Some users in similar situations have found it best to change providers.

[ Thread continues here (6 messages/9.36kB) ]



Talkback: Discuss this article with The Answer Gang

Published in Issue 156 of Linux Gazette, November 2008

Talkback

Talkback:124/smith.html

[ In reference to "Build a Six-headed, Six-user Linux System" in LG#124 ]

Chad Van Maele [chadvm at gmail.com]


Sun, 5 Oct 2008 15:40:13 +0000

I've tried and tried and I can't get this to work, I'm using Debian/etch, xorg 7.3, and xdm. When try to start the X server manually, seat0 works ok, but I've had varying success with seat1, sometimes it starts, but not on the same vt, so monitor0 goes black, or it will start on the same vt, but no window manager will start, the mouse and keyboard work ok though. I've looked at an exhaustive amount of other howto's, but most of them are outdated, saying I need Xephyr to do it. I've gone to irc.freenode.net and went to #xorg, but no one there even talks at all, just a bunch of IRC trolls I guess. I don't know where to look for help anymore, so I'm writing this email. I don't want to be a bother, but I just don't know what else to do. Thanks.

-- 
Chad Van Maele
http://www.tshirthell.com/store/link.php?id=RGlvbml4

[ Thread continues here (2 messages/2.20kB) ]


Talkback:155/xkcd.html

[ In reference to "XKCD" in LG#155 ]

Jimmy O'Regan [joregan at gmail.com]


Thu, 9 Oct 2008 18:22:43 +0100

XKCD Improving the Internet ... Yet Again http://idle.slashdot.org/article.pl?sid=08/10/09/1618235&from=rss

The real story is here: http://www.networkworld.com/community/node/33807


Talkback:155/lg_tips.html

[ In reference to "2-Cent Tips" in LG#155 ]

Jimmy O'Regan [joregan at gmail.com]


Fri, 10 Oct 2008 01:01:08 +0100

[[[ This is specifically a followup to the thread http://linuxgazette.net/155/misc/lg/two_cent_tip__download_whole_directory_as_zip_file.html -- Kat ]]]

2008/9/16 Silas S. Brown <ssb22@cam.ac.uk>:

>
> You shuld be able to use Gradint as well if you
> can find someone to record the words and
> phrases for you.

I wanted to mention this at the time, but I couldn't find the link: http://shtooka.net/ is a project aimed at just this idea - of providing pronunciations of words in various languages.

[ Thread continues here (5 messages/8.93kB) ]


Talkback:152/srinivasan.html

[ In reference to "A Slightly Advanced Introduction to Vim" in LG#152 ]

Eric Deschamps [erdesc at free.fr]


Tue, 30 Sep 2008 14:46:31 +0200

Hi,

I guess there is another error on this document.

In the section "Miscellaneous Commands"

		 Shift+j (or J) - This will brings up the previous line to the end of the current.

It should be : This brings up the next line to the end of the current, shouldn't it ?

Best Regards,

Eric

[ Thread continues here (2 messages/1.39kB) ]


Talkback: [LG 86] 2c Tips #20

Jimmy O'Regan [joregan at gmail.com]


Tue, 14 Oct 2008 09:33:58 +0100

[[[ This is clearly the wrong Talkback link, but after some serious searching of the LG archives, I can't find the source anywhere. It may be from some (rare) unpublished recent material. Lacking a better subject line, I've opted to retain the (inaccurate) original it was sent in as. -- Kat ]]]

2008/3/10 Ben Okopnik <ben@linuxgazette.net>:

> Oh, heck - the classic "The spirit is willing, but the flesh is
> weak"/"The vodka is good but the meat is rotten" problem is going to be
> with us for a long, long time.

Turns out that example is false, according to the MT Book (http://www.essex.ac.uk/linguistics/clmt/MTBook/):

There was/is an MT system which translated The spirit is willing,
but the flesh is weak into the Russian equivalent of The vodka is
good, but the steak is lousy, and hydraulic ram into the French
equivalent of water goat. MT is useless.
 
The `spirit is willing' story is amusing, and it really is a pity that
it is not true. However, like most MT `howlers' it is a fabrication.
In fact, for the most part, they were in circulation long before any
MT system could have produced them (variants of the `spirit is
willing' example can be found in the American press as early as 1956,
but sadly, there does not seem to have been an MT system in America
which could translate from English into Russian  until much more
recently --- for sound strategic reasons, work in the USA had
concentrated on the translation of Russian into English, not the other
way round). Of course, there are real MT howlers. Two of the nicest
are the translation of French  avocat (`advocate', `lawyer' or
`barrister') as avocado, and the translation of Les soldats sont dans
le café as The soldiers are in the coffee. However, they are not as
easy to find as the reader might think, and they certainly do not show
that MT is useless.

For our own ins, the previous release of our Catalan-English translator mistranslates 'Fidel Castro' (I requested that 'Faithful Castrate be the release name for 0.8.4, but was unsuccessful); the current versions of en-ca and en-es both mistranslate the Catalan and Spanish versins of 'International Monetary Fund' to something like 'Worldwide Bottom Fund' - which might be more accurate in the current climate, but still - which I fixed last week (you don't need to speak a language to work on MT, but it helps).



Talkback: Discuss this article with The Answer Gang

Published in Issue 156 of Linux Gazette, November 2008

News Bytes

By Deividson Luiz Okopnik

News Bytes

Contents:

Selected and Edited by Deividson Okopnik

Please submit your News Bytes items in plain text; other formats may be rejected without reading. (You have been warned!) A one- or two-paragraph summary plus a URL has a much higher chance of being published than an entire press release. Submit items to bytes@linuxgazette.net.


News in General

[lightning bolt]Linux Kernel 2.6.27.2 Launched

Version 2.6.27.2 of the Linux kernel was released on October 18th, introducing several bug-fixes to the 2.6.27 kernel that added a new file-system (UBIFS, optimized for pure flash-based storage devices), several new drivers, and Xen support for saving and restoring virtual machines,

The full changelog of the release can be found here: http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.27.2.

[lightning bolt]Mono 2.0 Release - Major Milestone Reached

Mono 2.0 was released on October 6th. This very specific release was much awaited by Mono users, and is a major milestone for the Mono project, by offering complete support for Microsoft's .NET 2.0 framework. This gives developers one more alternative for writing portable programs.

More details of the release:
- New controls for Windows.Forms namespace (Tool and Menu bars, DataGridView, and WebBrowser components are some of those components);
- Runtime performance improved;
- Updates to the C# compiler;
- LINQ and LINQ-to-XML support complete.
Along with other changes.

The full release notes can be found here: http://www.mono-project.com/Release_Notes_Mono_2.0.

[lightning bolt]Who says Open Source isn't about money?

Alan Lord wrote on the Inkscape discussion e-mail list:

This is a great story and Inkscape, along with lots of other FOSS, played its part...

Details on the design of a new Dutch coin at the link here: http://pythonide.blogspot.com/2008/10/how-to-make-money-with-free-software.html .

[lightning bolt]Away with the keyboards and mice!

TAG member Jimmy O'Regan sent in this news item, quoting from http://weblog.infoworld.com/fatalexception/archives/2008/10/microsofts_touc.html

"Most recently, Iron Man's Tony Stark designed CAD models for an entire suit of high-tech combat armor using only his bare hands. So how come we're all still using keyboards and ordinary old mice? InfoWorld's ace reporter Paul Krill was the first to spot a new project from Micr0s0ft Office Labs and Micr0s0ft developer Mike Wasserman that could bring those next-generation UIs one step closer."

The code is released under the MS-PL, which is open source and GPL-compatible:http://www.codeplex.com/touchless It's C#, so it should work with Mono (and if it doesn't, it's only a matter of time before some Mono person makes it work)

The article also mentions this project: http://www.hitl.washington.edu/artoolkit/ which is GPL and already works on Linux.

[lightning bolt]Which came first, the 3-D printer or the 3-D printer it made?

Jimmy O'Regan sent in another fantastic tidbit, this one about a self-replicating open source 3-D printer. "RepRap" stands for Replicating Rapid-prototyper, and it's simply too cool to explain briefly here. Code and specification are available under the GPL. "Not counting nuts and bolts RepRap can make 60% of its parts; the other parts are designed to be cheaply available everywhere." They use Art of Illusion ( http://www.artofillusion.org/ ) to design models.

Version 2.0 is being designed, with a goal of being able to reproduce its own electronics, too. They've also had success with plywood http://reprap.org/bin/view/Main/PlywoodRepRaps

Go directly to the RepRap home page here: http://reprap.org/bin/view/Main/WebHome


Conferences and Events

Dreamforce '08 SaaS Conference
November 2 - 5, San Francisco, CA
http://www.salesforce.com/dreamforce/DF08/
Storage Decisions 2008
November 4 - 6, San Francisco, CA
http://storagedecisions.techtarget.com/index.html
System-on-Chip (SoC) Conference 2008
November 5 - 6, Newport Beach, CA
http://www.SoCconference.com
OpenOffice.org Conference - OOoCon 2008
November 5 - 7, Beijing, China
http://marketing.openoffice.org/ooocon2008
USENIX Large Installation System Administration Conference (LISA '08)
November 9 - 14, San Diego, CA

For the past 20 years LISA has been the focal point for the global community of system and network administrators. This year LISA continues that tradition, featuring innovative tools and techniques essential for your professional and technical development. LISA is the premier forum for presenting new research in system administration. We selected papers showcasing state-of-the-art work on topics including configuration management, parallel systems deployment, virtualization, and security. Bring your perplexing technical questions to experts at LISA's Guru Is In sessions; explore the latest commercial innovations at the Vendor Exhibition; benefit from opportunities for peer interaction (a.k.a. the "Hallway Track").

LISA '08

Early registration discounts for LISA '08, taking place November 9–14, 2008, in San Diego, are now available. Register by Friday, October 17, and save up to $300!

Agile Development Practices 2008
November 10 - 13, Orlando, FL
http://www.sqe.com/AgileDevPractices/
ISPcon Fall 2008
November 11 - 13,San Jose, CA
http://www.ispcon.com/future/f08_schedule.php
Mashup Camp Silicon Valley 2008 - Free Registration
November 17 - 19, Mountain View, CA
http://www.mashupcamp.com/
QCon-SF 2008
November 17 - 21, San Francisco, CA
http://qconsf.com/sanfrancisco-2008
SANS 9th Annual Security Conference
November 19 - 21, Washington, DC
https://www.sans.org/secitconf08/
SpringOne Americas
December 1 - 4, Hollywood, FL
http://americas.springone.com
VMware Virtualization Forums
December 3 - San Diego, CA
December 11 - Boston, MA
http://info.vmware.com/content/VirtualizationForum_Home
USENIX Symposium on Operating Systems Design and Implementation (OSDI '08)
December 8 - 10, San Diego, CA

Join us at the Paradise Point Resort for this symposium covering the most innovative systems research, including over 20 high-quality papers in areas including OS architecture, cloud computing, and monitoring, as well as a poster session. Don't miss the opportunity to gather with researchers from across the systems community in what has become a premier forum for discussing the design, implementation, and implications of systems software.

OSDI '08

Register by November 17 and save $150!

http://www.usenix.org/osdi08/lgb


Distro News

[lightning bolt]Ubuntu 7.04 End of Support on October 19th

On October 19th, Ubuntu 7.04 reaches the end of its support life. Supported for 18 months, from April 19th 2007 to October 19th 2008, this version of the Ubuntu distribution will now stop receiving new security patches and updates.

The only direct update path for users of Ubuntu 7.04 is upgrading to Ubuntu 7.10. Details of the process can be found here: https://help.ubuntu.com/community/GutsyUpgrades.

[lightning bolt]Red Hat High Performance Computing (HPC) Solution Launched

Red Hat announced the launch of Red-Hat's HPC Solution, a package of tools, including the operating system, cluster-creation and management tools, and development tools, aimed at creation of an HPC cluster in a single, easy to install package.

Backed up by Red Hat support services, the solution brings creation of such services to a new level. According to Red Hat, "With the Red Hat HPC Solution, making changes to the cluster is a snap. With its elegant and powerful node group and repository manager, you can deploy new updates, patches, and software packaged as kits, or re-image new nodes in minutes with no interruption in uptime. Managing four hundred nodes requires as little effort as managing four nodes".

You can check more details of the solution here: http://www.redhat.com/hpc/

[lightning bolt]Ubuntu 8.10 Server Focuses on Java, Virtualization, and System Management updates

The Ubuntu 8.10 Server edition is shipping with several updates, some of then being significant improvements in Virtualization, Java and System Management capabilities.

For virtualization, Ubuntu 8.10 Server comes with a virtual machine builder, allowing the user to create and manage virtual machines via the command line, giving users the option to automate those tasks with simple scripts. It also comes with an updated KVM, allowing more-flexible memory management, including the ability to change a virtual machine's memory amount without the need to reboot it.

By fully supporting Apache Tomcat 6.0 and the Java OpenJDK, Ubuntu 8.10 server also gives developers more options for developing and deploying Java applications.

Ubuntu 8.10 Server also comes with several other improvements, including:
- ClamAV and SpamAssassin in the main repositories, for virus and spam detection;
- Ability to encrypt private directories;
- Landscape (Canonical's System Management Solution) integration within the installer;
- Improved RAID support.

More information about Ubuntu 8.10 Server can be found here: http://www.ubuntu.com/server;
And you can download it here: http://www.ubuntu.com/download;
For more information about Canonical's Landscape, follow this link: http://www.canonical.com/landscape.

[lightning bolt]Ubuntu 8.10 To Be Released on October 30th

Ubuntu's next release, codenamed Intrepid Ibex, will be released on October 30th, and will include several updates, and some new features as well. Some of those features are:

- Ability to install from an USB drive;
- New instant-messaging software;
- Guest Sessions (temporary, secure sessions, used when sharing your computer with someone else);
- 3g network support;
- Built-in time tracker.

Canonical also teamed up with BBC to ensure most BBC material will be available trough Ubuntu 8.10's default media players.

More information about the release can be found on http://www.ubuntu.com/, and downloads will be available on http://www.ubuntu.com/download.

[lightning bolt]Debian 4.0r5

The fifth revision of the popular Debian 4.0 ("Etch") distribution is available for download, containing several fixes for security problems of the stable release.

The full announcement of the release, including a list of fixes and update instructions, can be found here: http://www.debian.org/News/2008/20081023.

[lightning bolt]Linux Mint 5 Fluxbox Released

Linux Mint announced on October 21th the availability of the Linux Mint 5 "Elyssa", Fluxbox edition, developed to be a lightweight distribution, running the Fluxbox window manager.

New features includes an auto-generated and auto-updated Fluxbox menu, Thunar custom actions, better language and localization support, multimedia compatibility with several pre-installed codecs and plugins, and more.

More details and download links can be found here: http://www.linuxmint.com/blog/?p=404.

[lightning bolt]Foresight Linux 2.0.5

On October 17th, Foresight Linux 2.0.5 was released to the public, being the first stable distribution to feature the latest GNOME, 2.24. The distribution also contains several other cutting-edge updates, including OpenOffice.org 3.0 and X.Org server 1.5.1.

More details about this release and download links can be found on http://www.foresightlinux.org/releases/2-0-5.html.


Software and Product News

[lightning bolt]Java 6 Updated!

Sun's Java Standard Edition 6 Update 10's availability was announced. It can be downloaded at http://java.sun.com/javase/downloads/index.jsp. Changes include faster startup for Java applications, new Direct3D capabilities, several bug fixes, and more.

More information about the release can be found here: http://java.sun.com/javase/6/webnotes/6u10.html

[lightning bolt]Flash Player 10 Launched

Adobe launched Flash Player 10, with several updates and new features. This update, unlike Flash Player 9, was released on the same day for several platforms, including Linux, Windows, and Mac OS X.

While the release does not seem too important for Flash development on Linux (one of the main new features, the ability to use custom filters and effects, can only be achieved on Windows and Mac OS, as it requires the Adobe Pixel Bender toolkit, that runs only on those platforms), for those who want only to be able to watch Flash clips on Linux, this is still a release worthy of note.

[lightning bolt]Gnome 2.24

GNOME received a new update, this month, and includes several new features along with a long list of improvements and bug fixes.

The new features of this release include:
- Included instant-messaging solution;
- Ekiga 3 for audio/video conferencing;
- Tabs and new ways of viewing files on the file browser;
- New and improved screen resolution controls;
- Support for sound themes;
- Several accessibility updates;
And much, much more.

Full details can be found here: http://library.gnome.org/misc/release-notes/2.24/

[lightning bolt]VMWare Workstation 6.5 Released

VMWare Workstation, the virtualization solution from VMWare, just received a new version, and it's full of new features and bug fixes.

Some of those new features include:
- Support for new 32- and 64-bit operating systems;
- Unity mode (integrates the guest and the host desktops);
- Better internationalization support;
- Easy-install option for Linux and Windows guests;
- Virtual network editor for linux hosts;
- Several performance improvements (including network and USB speed-ups);
- Several new developer tools.

The full release-notes can be found here: http://www.vmware.com/support/ws65/doc/releasenotes_ws65.html



Talkback: Discuss this article with The Answer Gang


[BIO]

Deividson was born in União da Vitória, PR, Brazil, on 14/04/1984. He became interested in computing when he was still a kid, and started to code when he was 12 years old. He is a graduate in Information Systems and is finishing his specialization in Networks and Web Development. He codes in several languages, including C/C++/C#, PHP, Visual Basic, Object Pascal and others.

Deividson works in Porto União's Town Hall as a Computer Technician, and specializes in Web and Desktop system development, and Database/Network Maintenance.


Copyright © 2008, Deividson Luiz Okopnik. Released under the Open Publication License unless otherwise noted in the body of the article. Linux Gazette is not produced, sponsored, or endorsed by its prior host, SSC, Inc.

Published in Issue 156 of Linux Gazette, November 2008

A (not so) short overview of the Geographic Information System GRASS

By Matteo Dell'Omodarme and Giada Valle

The aim of this article is to introduce the GRASS GIS software and to show its basic usage. As with all GIS software, it is quite complex: there is a huge number of features and controls, and non-trivial theoretical information is required even for basic use. Nevertheless, we think that the quick introduction presented here could be of some help for neophyte users who want to master a GIS program.

We chose to devote a large part of this article to practical use of the software, in particular to real data gathering, reprojection of maps, and output of results. The decision is motivated by the fact that it is very difficult (and in some case impossible) to find this information systematically presented on the Net. Conversely, the introduction to the interface and the ancillary theoretical information are touched upon only briefly, because they can be retrieved from many sources (see the section at the end of this document.)

Since there are so many topics to introduce, and since we do not want to sacrifice clarity to brevity, some topics (i.e., climatological map-making, interfacing GRASS with statistical software, etc.) are postponed until another article.

Introduction

A Geographic Information System (GIS) is software for gathering, storing, managing, and presenting data that are geographically referenced. The characteristic of a GIS is the capability of examining different sets of geo-referenced data and reaching a conclusion about the existing relationship among the data. In other words, a GIS can link information that is difficult to associate by other means.

For example, it is possible to relate some geo-referenced information (e.g., disease foci) with other data (e.g., location of urbanized and rural zones, climatological data). This can reveal new information that can lead to better decisions about attempting to control the disease's spread.

Nowadays, GIS software is used in several scientific fields, including resource management, environmental impact assessment, ecology, archaeology, cartography, and epidemiology.

GISes are normally capable of different types of spatial analysis and geographical feature processing. Some examples:

Geographic Resources Analysis Support System (GRASS) GIS (http://grass.osgeo.org/) is a geographic information system released under the GNU General Public License (GPL). GRASS is being developed for the GNU/Linux, Mac OS/X, and Windows operating systems. GRASS development is split into stable and development branches. The first one is recommended for most users.

GRASS is capable of data management, image processing, graphics production, spatial modeling, and visualization of many types of data. It's an official project of the Open Source Geospatial Foundation (http://www.osgeo.org/).

GRASS was originally developed by the U.S. Army Construction Engineering Research Laboratories (USA-CERL) as a tool for land management and environmental planning by the military. From 1982 to 1995, USA-CERL led the development of GRASS, with the involvement of numerous others, including universities and other federal agencies.

In 1995, the software's development was taken over by a group formed at Baylor University. In this period, the package was ported to Linux. In 1998, Markus Neteler, current project leader, announced the release of GRASS 4.2.1, offering major improvements including a new graphical user interface. In October 1999, the license of the originally public-domain GRASS software was changed to GNU GPL (specifically, starting with GRASS v. 5.0). Users interested in additional historical background can look at http://grass.osgeo.org/devel/grasshist.html.

GRASS has now evolved into a powerful utility, with a wide range of applications. It is currently used in academic institutions and commercial companies around the world, as well as many governmental agencies and environmental consulting companies.

1.1 Basic features and documentation

Upon starting GRASS, the user is presented with a Unix shell containing a modified environment supporting execution of GRASS commands (known as modules). Users can also interface with the majority of GRASS modules and capabilities through a graphical user interface (GUI). There are over 200 core GRASS modules included, and over 100 add-on modules contributed by users and offered on the GRASS Web site. The GRASS libraries and core modules are in C, while other modules are in C, Unix shell, Tcl, or other scripting languages.

Data managed by GRASS can be roughly divided into geometric and attribute data. In the first case, there are:

Attribute data are attributes interconnected with the data types mentioned above. For example, a vector layer representing countries boundaries can be associated with data concerning state population, country identification code, etc. Data are usually saved within GIS or in a database system coupled with GIS.

GRASS supports an extensive range of raster and vector formats through binding to GDAL/OGR libraries. GRASS release 6 introduces a new topological 2D/3D vector engine, and support for vector network analysis. Attributes are usually managed in .dbf database files, although other DBMSes can be used. The system is capable of visualizing 3D vector graphics data.

A huge amount of information, documentation, and other resources are available for GRASS. The documentation includes a very useful internal help system, a FAQ list, and a number of textbooks. In addition to the textbooks (http://grass.osgeo.org/gdp/books.php), the GRASS documentation Web page also hosts various user-contributed documentation, such as tutorials and courses (http://grass.osgeo.org/gdp/tutorials.php) as well as manuals (http://grass.osgeo.org/gdp/manuals.php).

Other GIS software is available, both open source and proprietary. Among the former, GRASS is the most widely used, especially for academic research. As for commercial products, the ArcInfo workstation from ESRI and Mapinfo Professional from PitneyBowes Mapinfo are the market leaders. The license price is 9,000-15,000 USD (depending on features) for ArcInfo and about 1,700 USD for Mapinfo. Both programs have been developed for Windows, GNU/Linux, and Unix operating systems but lack support for Mac OS/X.

1.2 Installing GRASS

The latest version can be found at the GRASS download page. Here, you can find precompiled binaries for several platforms as well as the source code.

Installation from source requires at least two extra libraries: PROJ4 for management of projections and GDAL/OGR for reading and writing various GIS data formats. These libraries must be installed in the right order, before installing GRASS.

The first step is to download PROJ4 from http://trac.osgeo.org/proj/. The latest version of the library is 4.6.0. The installation procedure is straightforward:

tar zxf proj-4.6.0.tar.gz
cd proj-4.6.0/
./configure
make
make install

This library allows management of map projections, which are any method used in cartography to represent the two-dimensional curved surface of the Earth or another body on a plane. Flat maps could not exist without map projections, because a sphere cannot be laid flat over a plane without distortions. To run a GIS package, some details about this topic are required. Some basic elements are given in the following:

The second step is installing the GDAL/OGR library combination. The latest version (1.5.2) can be downloaded from http://download.osgeo.org/gdal. The install procedure is, again, the usual one:

tar zxf gdal-1.5.2.tar.gz
cd gdal-1.5.2/
./configure
make
make install

GDAL is a translator library for raster geospatial data, while the OGR library (which lives within the GDAL source tree) provides a similar capability for simple-features vector data. These libraries allow import and exchange of files between different GIS programs.

Finally, GRASS can be installed. The last stable version of the software (currently 6.2.3) is located at the GRASS download page. The software is installed by the now-familiar method:

tar zxf grass-6.2.3.tar.gz
cd grass-6.2.3/
./configure
make
make install

1.3 Running GRASS

As a first step, the user is required to create a directory (GRASS database) that will hold the GRASS data. A usual choice is to create a directory named 'GRASS' (or 'grassdata') in the user's HOME directory.

A subdirectory tree, named location, is automatically created in the GRASS database for each defined project region. All project data are saved in the location subdirectory. The location can be further subdivided into map subdirectories called mapsets.

All general information about the project area is stored within the location in the PERMANENT mapset, which is automatically generated by GRASS. A user can create one or several mapsets for a given location. They can have the extent of the whole project or smaller. This characteristic makes it possible to have several users working on one project at the same time, without the risk of destroying each other's data. Other information about the tree structure of a location can be found, for example, at http://www.gdf-hannover.de/lit_html/grass60_v1.2_en/node10.html.

GRASS can then be started by typing the following at the shell prompt:

grass62

The program opens a modified shell and displays the following text:

WELCOME TO GRASS              Version 6.2.3         2007

   1) Have at your side all available GRASS tutorials

   2) When working on your location, the following materials
      are extremely useful:
      - A topo map of your area
      - Current catalog of available computer maps

   3) Check the GRASS Web pages for feedback mailing lists and more:
      http://grass.itc.it/
      http://grass.ibiblio.edu/

Hit RETURN to continue

After the user presses Enter, GRASS displays a graphical window from which one can choose the location and the mapset to open, as in the following figure:

GRASS

From this window, one can read two important pieces of data:

Besides choosing a defined location, it is also possible to create a new location in three different ways. These methods differ in the amounts of information required of the user. The simplest case occurs when data about the location are self-contained in the imported map (geo-referenced file); in these cases, no other information is needed.

The definition of a new location requires input (directly by the user or indirectly from a map) of the following information:

These settings are critical for correct display and management of the location. Therefore, before going further, it's worthwhile to introduce some theoretical information about projections and coordinate systems.

1.4 Excursus: projections and coordinate systems

In a GIS, the information is stored with coordinates; therefore, a coordinate system has to be specified. For geographical data, the geographic system (the well known latitude and longitude) or a geodetic projection like Universal Transverse Mercator (UTM) or Gauss-Krüger are the usual choices.

The UTM coordinate system is a grid-based method of specifying locations on the surface of the Earth. It differs from the traditional method of latitude and longitude in several respects. The Gauss-Krüger system is similar to the UTM system with only small differences. More details on these coordinate systems can be found in further readings listed in the final section.

To optimize the projection from the geoid into a flat map, the Earth is approximated by an ellipsoid.

Different projections are used because each projection is particularly appropriate for certain uses or areas. For example, a projection that accurately represents the shapes of the continents will distort their relative sizes. GRASS offers predefined projections, as well as the geographic system and the possibility to define a geodetic projection by yourself. In this last case, the projection type, the ellipsoid and the datum (a reference from which measurements are made) for referencing the ellipsoid have to be specified.

1.5 Creating a location: the quest for data

The best way to appreciate the functionality of GRASS is to see it in action on a real case-study. As a first step, let us introduce the usual question that worries every new user: "Well, I have just installed this wonderful software, and I have some geo-referenced data to analyze and relate to geophysical characteristics of my research area. But what about geophysical maps? Where can I get them? What kinds of thematic data are available for my region of interest?"

The answer obviously depends on the analysis you have planned, and on the geographic region of interest. Some frequently desired thematic maps are:

Let us state clearly that, if you are interested in a very specific and very high resolution map, your only hope is to find this resource for a fee (usually substantially high) from a local agency. Sometimes, these data are distributed in electronic format; otherwise, you have to scan them from the hardcopy, yourself.

If you are instead interested in large-region medium resolution (30 - 100 m) maps, the Net hosts several amazing resources you should use. In the following section, we describe Net sources where this information is available for free download, and how to import them into GRASS.

2. Elevation data

The Shuttle Radar Topography Mission (SRTM) obtained elevation data on a near-global scale to generate the most complete high-resolution digital topographic database of Earth. SRTM consisted of a specially modified radar system flown onboard the Space Shuttle Endeavor during an 11-day mission in February of 2000. The SRTM data are available as 3 arc second (90m resolution). 1 arc second data are available for US. The Global Land Cover Facility provides free access to the data (http://glcfapp.umiacs.umd.edu:8080/esdi/).) The data contains "holes" where water or heavy shadow prevented the evaluation of elevation.

In the following example we show how to obtain elevation data for Tuscany (Central Italy), where we currently live.

Data gathering starts by connecting to the Global Land Cover Facility Web page. In the page, the three ways in which you can access the data (Map Search, Path/Row Search, Product Search) are displayed. Let us choose the first, Map search: the user must choose the desired product in the left panel and select the world region of interest (map on the right). Since we are interested in elevation data, we can choose one among the four available options. Note that GTOPO30 is a global digital elevation model with a horizontal grid spacing of 30 arc seconds, so about ten times less accurate than SRTM data. The first two options are different ways to access the SRTM dataset: the first one allows download of tiles of 1 degree square, while the second accesses WRS-2 tiles to approximate Landsat scenes, which we will introduce in the next paragraph. For reasons that will become clear, we choose to select this second option.

Now, we have to select the geographic region of interest, by clicking on the place panel, just above the world map. In the displayed text box type "Toscana, Italy" and press Enter. (In general, you type "Location, State".) You should then see a page similar to the one displayed in the figure.

srtm-1

There are 35 maps available in the selection. The next step is to select the ones of interest. Having pressed the button "Preview & Download", the selection window is displayed. It looks like the figure below:

srtm-2

The screen is divided into two zones. In the lower one available maps are listed, with their associated information. In particular, you can see the type of data (GeoTIFF format, a widely used raster format), the status (unfinished, unfilled, finished-a, and finished-b in ascending order of processing level and of our choice). In the third column, there are data which allow the identification of path/row attributes, used for Landsat satellite data. The user can select a map by clicking on its ID (first column). In the upper area, information about the selected map is shown. In the right part, the user can check which zone is covered by the map. This picture can guide the selection of really interesting maps. After identification of a desired map, the user should press the download button, and download the .tif.gz file from the window that will open.

At the end of the scan, you will find that only four maps are required for download: ID 180-332 (path/row: 191/030), ID 180-364 (192/029), ID 180-365 (192/030), and ID 180-408 (193/029). These files can be saved in a directory, e.g., $HOME/SRTM, and uncompressed.

2.1 Importing a raster map

GeoTIFF files incorporate information about the coordinate system, the projection, and the datum used for their georeferencing. In this way, it is possible to use them to define a new GRASS location, and subsequently import them into the software.

The process is a little tedious. From the start page of GRASS (displayed in the first figure of this article), define a new location by pressing the button "Georeferenced file". In the window that will open, insert the name of the new location (e.g., "tuscany"), and, in the last one, select the path to one of the four downloaded files (e.g., SRTM_ffB03_p192r029.tif). End the location definition by pressing "Define location". GRASS generates the tree for the new location, and displays a warning requesting to close the program.

At the restart, you will be able to see your first new generated location!

Select the location tuscany, and the mapset PERMANENT, then click on "Enter GRASS". The shell will display the following greeting message:

Welcome to GRASS 6.2.3 (2007) 
GRASS homepage:                          http://grass.itc.it/
This version running thru:               TC Shell (/bin/tcsh)
Help is available with the command:      g.manual -i
See the licence terms with:              g.version -c
If required, restart the graphical user interface with: gis.m &
When ready to quit enter:                exit

Mapset <PERMANENT> in Location <tuscany>
GRASS 6.2.3 > 

Besides that, three more windows will open: the GUI (window GRASS 6.2.3 GIS Manager), the output window which will host the output of all commands issued by means of GUI, and the map display window.
At this point, the Tuscany location is created, but it hosts no data. Importation of the four downloaded maps can be performed either by shell command or by GUI. We illustrate first this second approach, since it will also show the shell command associated with the GUI procedure.

From the GUI menu bar, select File / Import / Raster map / Multiple formats using GDAL. Fill in the form that will open, as in the following figure:

r-in-gdal

The only required fields are the name of input file and the output name, i.e., the internal name GRASS associates to this map. The box "Extend location extents based on new datasat" will extend the location to encompass the imported map.

In the window's last line, the shell command associated with the import operation gets displayed. Its structure is really simple, and makes it faster to import the other maps from the shell. At the shell prompt, insert these commands:

r.in.gdal -e input=/home/matt/SRTM/SRTM_ffB03_p192r030.tif output=192_030
r.in.gdal -e input=/home/matt/SRTM/SRTM_ffB03_p193r029.tif output=193_029

Unfortunately, trying to import the fourth map, you receive an error:

r.in.gdal -e input=/home/matt/SRTM/SRTM_ffB03_p191r030.tif output=191_030

ERROR: Projection of dataset does not appear to match current location.

What is happening? The answer is simple, as is the solution. The projection information associated with this map is different from the those of the three imported maps. The simpler solution is to import the map in a new location (say, "tuscany2"), which will be created ad-hoc, and then reproject the data in the current location. The first step is accomplished by adding a location parameter to r.in.gdal:

r.in.gdal -e input=/home/matt/SRTM/SRTM_ffB03_p191r030.tif output=191_030 location=tuscany2

Since a map's projection does not extend the region boundaries on the fly, you have to be certain that your region is large enough to contain the projected map. In this case, this is false, because the fourth map is partially outside the east boundary. You have to manually adjust this boundary with the command g.region (Config / Region / Change region settings from GUI), and then reproject (Raster / Develop map / Reproject raster from other location from GUI).

Which value has to be set as the east boundary? To make an educated guess, it is better to look at the coordinates of one of the imported maps. You can do it by setting the boundaries of the location to match one of the maps and then extrapolate where the fourth map will be located. For example:

g.region rast=192_029

g.region -p

The first command sets the location's boundaries to those of map 192_029. The second one queries all of the location's parameters; from its output, we can learn several things, such as coordinate system, projection, datum, and coordinates of the boundaries (Config / Region / Display region settings from GUI):

projection: 1 (UTM)
zone:       32
datum:      wgs84
ellipsoid:  a=6378137 es=0.00669437999014138
north:      5065399.56462278
south:      4821139.56462278
west:       576236.63839707
east:       828866.63839707
nsres:      90
ewres:      90
rows:       2714
cols:       2807
cells:      7618198

In the output above, the measures are in meters. Now it is possible to set the eastern boundary:

g.region e=968866

And reproject the map in the current location:

r.proj input=191_030 location=tuscany2 output=191_030 method=nearest

The syntax of r.proj is self explanatory. The option method chose the interpolation algorithm to be used. (Note: if you set an incorrect east boundary, and you have to reproject the map, you must add the flag --overwrite at the end of command r.proj.)

2.2 Displaying and patching raster maps

It is time to see some output from GRASS! Let us display one of the imported maps, say, 192_029. This is done by using the GUI and pressing the "Add raster layer" button GUI-rast.

A new empty raster layer is then added for visualization in the upper part of the GUI window. By clicking on the name of the layer, you cause the lower part of the window to display a form. Here, you can define which map is associated with the layer. Press on the Base map button, and select the map from the list that is then shown. At the end of the procedure, the GUI should look like this:

GUI-layer-rast

You should now admire the results of your work. In the Map display window, press the "Display active layer" button (the first one). The window will probably remain empty. In this case, we have to set the zoom of the displaying window to the one of the region boundaries (which we set in the previous paragraph). You should press on the button "Zoom to..." zoom, and select the option "Zoom to current region" as shown in the figure:

GUI-rast

The displayed map should look almost red, with the exception of a few yellow areas, which correspond to the no-data zones. A useful instrument available from the Map Display window is the query button query. Select it and then press over the map; the value associated to the cell is shown in the output window. You will discover that, for example, no-data areas are represented by value -32768.

Now it's map-cleaning time! We have to patch the four maps to obtain a new composite map. Before that, we have to clean up the maps. In fact, GRASS expects that no data zones will be identified by a NULL value, while, as we know, SRTM maps identifies no-data areas with the code -32768. The following commands correct this problem (Raster / Develop map / Manage null values from GUI):

r.null map=191_030 setnull=-32768
r.null map=192_029 setnull=-32768
r.null map=192_030 setnull=-32768
r.null map=193_029 setnull=-32768

The next step is definition of the correct location boundaries. GRASS can make boundaries from a series of maps. We use this characteristic by issuing the command:

g.region rast=191_030,192_029,192_030,193_029

The resulting boundaries just encompass the four maps.

As our last step, we can merge the maps, and define an appropriate colormap:

r.patch in=191_030,192_029,192_030,193_029 output=elev
r.colors map=elev rules=srtm

The command r.patch (Raster / Overlay maps / Patch maps from GUI) creates a new map, named elev, by merging the four existing ones, while r.colors (Raster / Manage map colors / Set colors to predefined color tables from GUI) defines a colormap using an appropriate predefined ruleset (srtm).

To correctly visualize the map, we have to restart GRASS after having closed the session by entering exit at the shell prompt, and closing the window manager. After having restarted the software, we can display the final result of our work: repeat the procedure used for displaying a raster layer and select the map elev. The map should look like this one:

tuscany

If the result is satisfactory, you can delete the four small maps. From the GUI, the path is "File / Manage maps and volumes / Remove maps" and fill the raster field with the name of the maps to delete. From the shell, type:

g.remove rast=191_030,192_029,192_030,193_029

At the end of this section, we have created an elevation map for the Central Italy area. We will later extract from it the data related to the Tuscany region, and discard the rest.

As a final note, since the coverage of SRTM data is worldwide, the procedure can be repeated for any location.

3. Landsat data

Landsat-7 is a satellite launched on April 1999. It has sun-synchronous, near-polar orbit that allows it to record the surface of the Earth in a pattern of overlapping 185 km swaths, completing one scan every 233 orbits. The pattern is catalogued as a tiled global grid of 233 paths and 248 rows in the so called Worldwide Referencing System (WRS). We have seen an example of this reference system when downloading SRTM data.

The satellite is equipped with an instrument called the Enhanced Thematic Mapper (ETM+), a multi-spectral radiometric sensor. The sensor records eight bands of data with varying spectral and spatial resolutions (30 m spatial resolution for red, green, blue, near infrared, and two bands of medium infrared; 60 m for thermal infrared; and a 15 m panchromatic band). A list of the bands with their spanned wavelength (in micrometers) is given below:

  1. Blue-Green: 0.45-0.515 μm
  2. Green: 0.525-0.605 μm
  3. Red: 0.63-0.690 μm
  4. Near infrared: 0.760-0.900 μm
  5. Medium infrared: 1.550-1.750 μm
  6. Thermal: 10.40-12.5 μm
  7. Medium infrared: 2.080-2.35 μm
  8. Panchromatic: 0.52-0.92 μm

Landsat data are available from many sources. During April 2008, all data gathered by the satellite were released into the public domain, so it is possible to download scenes acquired in different months and different years. (See http://landsat.usgs.gov/products_data_at_no_charge.php for details.) We will show here a simpler method of download, limited at one dataset acquired in the years 2000-2001, available at the Global Land Cover Facility Web page. Data gathering procedure is identical to that described in the previous section. The user must select the right product by checking the ETM+ box in the interface shown in second figure of this document.

The result should be similar to the figure below:

landsat

The same path/row maps downloaded for elevation data are required; this correspondence greatly accelerates the user choice. In this case, the map IDs are: ID 036-387, ID 036-429, ID 036-430, and ID 036-128.

By pressing the download button, you can see that the data are available as a set of eight GeoTIFF files (one for each ETM+ band). They are much bigger than SRTM data, since the resolution is about three times as great. You can download whatever band you desire.

Landsat data can be used to assess the type, extent, and condition of vegetation over a region, or vegetative change over time. These studies require that a vegetative index be calculated from sets of remotely-sensed data. The most widely used index is the Normalized Difference Vegetative Index (NDVI). The NDVI is calculated as a ratio between measured reflectivity in the red and near-infrared portions of the electromagnetic spectrum. These two spectral bands are chosen because they are affected by the absorption of chlorophyll in leafy green vegetation and by the density of the green vegetation on the surface. Moreover, in red and near-infrared bands, the contrast between vegetation and soil is at a maximum.

ETM+ bands 3 and 4 provide red and near-infrared measurements, and therefore can be used to generate NDVI data sets with the following formula:

NDVI = (Band 4 - Band 3) / (Band 4 + Band 3)

The NDVI equation produces values in the range of -1.0 to 1.0. Vegetated areas will typically have values greater than zero, and negative values indicate non-vegetated surface features such as barren, water, ice, snow, or clouds. NDVI value are usually scaled for display, as is the following equation:

Scaled NDVI = 100 (NDVI + 1)

3.1 Making a raster map from existing ones

To evaluate the NDVI index for Tuscany, we have to download bands 3 and 4. These files can be saved in a directory, say, $HOME/Landsat, and uncompressed.

In order to preserve the projection values of Landsat data, the data can then be imported in a new location, e.g., landsat, in order to preserve the projection values of Landsat data. As shown in the preceding section, the command (dataset 193/29, band 4) is:

r.in.gdal -e input=/home/matt/GIS/Landsat/p193r029_7t20010801_z32_nn40.tif output=193_29_4 location=landsat

Restart GRASS, enter the new defined location, mapset PERMANENT, and finish the import:

r.in.gdal -e input=/home/matt/GIS/Landsat/p193r029_7t20010801_z32_nn30.tif output=193_29_3

r.in.gdal -e input=/home/matt/GIS/Landsat/p192r029_7t20000620_z32_nn40.tif output=192_29_4
r.in.gdal -e input=/home/matt/GIS/Landsat/p192r029_7t20000620_z32_nn30.tif output=192_29_3

r.in.gdal -e input=/home/matt/GIS/Landsat/p192r030_7t20000620_z32_nn40.tif output=192_30_4
r.in.gdal -e input=/home/matt/GIS/Landsat/p192r030_7t20000620_z32_nn30.tif output=192_30_3

As seen previously, map 191/30 has a different projection, so you have to read that map in a new location (one for each map), extend the region boundaries, and reproject the maps in the current location:

r.in.gdal -e input=/home/matt/GIS/Landsat/p191r030_7t20010803_z33_nn40.tif output=191_30_4 location=landsat2
r.in.gdal -e input=/home/matt/GIS/Landsat/p191r030_7t20010803_z33_nn30.tif output=191_30_3 location=landsat2b

g.region e=968866

r.proj input=191_30_4 location=landsat2 output=191_30_4 method=nearest 
r.proj input=191_30_3 location=landsat2b output=191_30_3 method=nearest 

As a last step, we can patch the maps together. Since, in this case, null values are represented by zeroes we can skip the r.null call by adding a -z flag to r.patch:

r.patch -z in=192_29_4,192_30_4,193_29_4,191_30_4 output=Landsat_B4
r.patch -z in=192_29_3,192_30_3,193_29_3,191_30_3 output=Landsat_B3

Now, all the elements for NDVI calculation are on hand. It is time to learn how to make a new map starting from existing ones!

The best approach is to use the shell command r.mapcalc:

r.mapcalc "NDVI=(float(Landsat_B4 - Landsat_B3) / (Landsat_B4 + Landsat_B3) + 1)*100"

To obtain a floating point division (the Landsat maps are integer maps), the float function is required. The command makes a new map, called NDVI, by operating, cell by cell, on the values in maps Landsat_B3 and Landsat_B4.

Finally, you should reproject the NDVI map into the tuscany location. To do that, exit from GRASS session, restart the program, and login into the tuscany location. Then issue the command:

r.proj input=NDVI location=landsat output=NDVI

If no problem arises, you will have two raster maps in this location. To recover disk space, all other locations can be erased. This can be done by exiting GRASS, changing directory to $HOME/GRASS, and deleting the directories tuscany2, landsat, landsat2, and landsat2b. The resulting map will be shown in the next section, once having learned how to "mask" it in order to show only the pixels of Tuscany.

As for SRTM data, Landsat data also is worldwide, so you can repeat the procedure for any location on the Earth.

4. Getting state and districts' boundaries

States', districts' and municipalities' boundaries can be retrieved from several sources, depending on the region of interest. In the case of Italian regions, for example, these are available from the ISTAT Web site (in Italian). The file of interest (regioni) can be downloaded from the right panel. This file contains Italy's regional boundaries.

4.1 Managing a vector map

Download the file into a new directory, e.g., $HOME/IT, then unzip it. The content of the archive consists of several files, one of which is a .shp file, a widely used vector type of data.

To read this type of data into GRASS, proceed in this way. Start GRASS in the tuscany location, then issue the command (File / Import / Vector map / Various formats using OGR from GUI):

v.in.ogr -e dsn=/home/matt/IT/reg2001_s.shp output=reg_boundaries min_area=0.0001 snap=-1 location=italy_reg

The import requires the specification of a datum transformation parameter. At the prompt, you can select parameter 8, valid for whole European area. At the end, you will have a new location, named italy_reg, which will contain a vector map named reg_boundaries.

Now, logout from the GRASS session and restart in the newly created location. The acquired vector map contains political boundaries of all the Italian regions. For our example, only the Tuscany region is of interest, so we have to extract an element from a map. For a vector map, this is a quite simple operation, since it is equivalent to extracting records from a database through a query.

To do the extraction, we should first learn how to inspect the database associated with a vector map. This can be easily done in the GUI. Vector layers are added to the display layer, in a fashion similar to the raster layers. The only change is that users have to press the "Add vector layer" button GUI-vect.

Select the added layer in the top part of the GUI window. In the bottom part, which will become available, choose the vector map you want to display, using the "Vector map" text field or the corresponding button. There are many more options for vector maps than for raster maps. For now, we are interested in the two buttons located at the end of the list. Scroll down the window, and you should see the buttons "show attribute columns" and "show attribute data".

GUI-vector

These buttons access the database linked to the map and return, respectively, the names of the database columns and the contents of the whole database, as shown in the next figure where these outputs are displayed. Note that each output is prefaced by the command forwarded from the GUI to the shell interpreter.

vector-db

The database contains four columns: a key used to index the table, the code of the regions, their names and their populations as recorded in 2001. The region code could be used for extraction of Tuscany. You can do that typing the command:

v.extract in=reg_boundaries out=tuscany_boundaries new=-1 where="COD_REG=9"

The map tuscany_boundaries will be created. The option new=-1 preserves the database table, exporting it in the new map.

4.2 Vector to raster conversion

Before exiting and connecting again in the tuscany location, we'll demonstrate how to convert a vector map to a raster one. This step will create a silhouette of the Tuscany region, which will later help us to "cut" the map of elevation and NDVI to the required extent. The resolution of the vector map is much lower than the one of the tuscany location (check it yourself: g.region -p). So, we increase the resolution and then convert the vector map:

g.region nsres=90
g.region ewres=90

v.to.rast input=tuscany_boundaries output=MASK use=val

This command makes the map MASK (a reserved map name), assigning to all cells the same value (option use=val). When a raster map named MASK exists in a location, most raster operations (on any raster map) will take place only in the grid cells specified by the MASK.

Now, restart GRASS, and access the tuscany location. Then, reproject the vector and the raster maps from italy_reg:

v.proj input=tuscany_boundaries location=italy_reg output=tuscany_boundaries

r.proj input=MASK location=italy_reg output=MASK

If we try to display the elev raster map, as we did before, the result is different. The elev map is masked, and only cells from Tuscany will be shown. The same behaviour occurs for NDVI map. The two maps are shown in the following figure (NDVI on the left, elevation on the right).

NDVI-elev

5. Land cover map

Land cover data are available from different sources, depending on the region of your interest. For European countries, you can refer to Corine Land Cover (CLC). CLC is a map of the European environmental landscape based on interpretation of satellite images; it provides comparable digital maps of land cover for each country for much of Europe, and includes 47 classes of land cover. Data on land cover is necessary for environmental policy, regional development, and agriculture policies.

Corine Land Cover 2000 data can be downloaded at 100 m horizontal resolution in raster format (GeoTIFF) from the European Environmental Agency Web site (http://dataservice.eea.europa.eu/dataservice/metadetails.asp?id=1007). Users are required to complete the "Agreement form for Corine Land Cover 2000", which is shown before download. In this way, you can download the whole European area contained in the file lceugr100_00EEA17938I.zip (approx 66MB). The download page is shown in the following figure. In the same page, you will find the file (four formats are available) containing the legend for interpretation of the map.

corine-download

There is also the possibility of downloading data in vector format (shapefile .shp), selecting on a grid only the region of interest (see http://www.eea.europa.eu/themes/landuse/clc-download).

After having downloaded the file into a directory, e.g., $HOME/CLC, uncompress it; the files will be expanded in a subdirectory named lceugr100_00. Then start GRASS in the tuscany location and import the data into the new location (e.g., CLC):

r.in.gdal -e input=/home/matt/CLC/lceugr100_00/lceugr100_00_pct.tif output=CLC location=CLC

CLC data can be reprojected in the current location:

r.proj input=CLC location=CLC output=CLC method=nearest

The new raster map can be displayed as shown before. The result is in the following figure; the presence of the raster MASK will cut the map at the desired boundaries.

corine

The various areas are depicted in different colors. The legend for interpretation of the map can be found in the file $HOME/CLC/lceugr100_00/clc_legend.xls. From this file you can see that, for example, CLC code 111 represents Artificial surfaces/Urban fabric/Continuous urban fabric and is depicted in RGB color 230-0-77, while code 221 is Agricultural areas/Permanent crops/Vineyards, and its RGB color is 230-128-0.

After a successful import of the map, you can recover a lot of disk space by erasing the location CLC. This is done by exiting GRASS and deleting the directory $HOME/GRASS/CLC.

5.1 Land cover map of US

When the research area is within the USA, the user can refer to the National Land Cover Database (NLCD 2001). It is an update of NLCD 1992, and includes 21 classes of land cover at 30 m cell resolution. The 76 standard NLCD mapping zones have been grouped into 15 larger zonal areas for easier distribution and download. The download site address is http://www.mrlc.gov/nlcd_multizone_map.php.

USA landcover

This page also contains a link to a zipped shapefile (3 MB) with the standard NLCD zones as well as a multizone attribute. Download it into $HOME/US directory, and unzip its content: a subdirectory named landcover_bndry_030607_shp will be created in the process. There, along with several other files, you'll find the landcover_bndry_030607.shp shapefile, which we'll later import into a new location.

The next step is to download the landcover for the zone of interest: you have the option to download data for a state or a superzone. As an example, let us choose the data for superzone 13 by clicking on it over the displayed US map. In the page that will open, select the link Land Cover zip file, and save the associated file area_13_landcover.zip (130 MB) in the directory $HOME/US, then unzip its content. Among other files, you will see the raster file landcover13_3k_022007.img, which we are going to import into GRASS.

After restarting GRASS in the tuscany location, we can import the superzone 13 raster data into a new location, i.e., USA:

r.in.gdal -e input=/home/matt/US/landcover13_3k_022007.img output=area13 location=USA

The import requires specifying the datum transformation parameter. At the prompt, you can select parameter 6, valid for whole North American region.

Exit GRASS, and restart in the USA location. Then, import the shapefile containing the states' boundaries, and extend the location boundaries to match the ones of the vector map:

v.in.ogr -o dsn=/home/matt/US/landcover_bndry_030607.shp output=boundaries min_area=0.0001 snap=-1

g.region vect=boundaries

Finally, we can display the resulting map, superimposing the vector layer to the raster one. To do that, in the GUI, you can start by adding a raster layer for visualization and selecting the raster map area13. Then, add a vector layer and select map boundaries. Using the vector layer, you can display a variety of objects: points, lines, centroids, and areas. To display only the boundary lines, deselect checkbox centroids and areas, as in the figure below.

USA layers

Then, activate the Map Display window, and press the display button. The zoomed result should look like this:

USA landcover A13

The described procedure could be adapted to other locations over the planet. A useful reference for landcover data is http://landcover.usgs.gov/landcoverdata.php.

6. Climatological data

Climatological data open a entirely new way of searching for information. Whereas elevation data can be considered static (i.e., their value does not change with time), and Landsat and land use data evolve slowly in time (in fact, you can find research papers comparing the Landsat situation in various decades to trace the global climate evolution), climatological data change rapidly. So, the first question you have to ask yourself is: "What type of data am I searching for?" You might be interested in daily or even hourly measurements, or in monthly means, or even in seasonal or yearly means. Moreover, yearly and seasonal means can be evaluated over a long period (say, 15 years) to smooth out the fluctuations introduced by a very hot summer or by a very cold winter.

The second aspect to be considered is the format in which data are distributed. The measurements are usually provided in tabular format: the tables contain the values recorded by a set of meteorological stations distributed over the region of interest. The main variables recorded are temperature (minimum, maximum, mean), humidity, and rainfall. Values in places without meteorological stations can be obtained by the user by mean of interpolation techniques. This is a difficult process, and requires a deeper analysis, because it involves either data manipulation (e.g., transformation of daily data in the tables to obtain a seasonal mean; selection of the interesting stations, etc.) or interpolation with advanced statistical techniques (e.g. universal kriging interpolation.) This last step can be performed by, for example, interfacing GRASS with R, a statistical program which we briefly described in issue 138 of the Linux Gazette.

Since a detailed description of all these depicted procedures would be long and complex, we'll postpone it until another article.

Moving further

In this long excursus, we have introduced some basic features of the software, but have neglected almost completely to describe the way in which data could be analyzed and related to each other. This approach is motivated by our experience with GRASS. In fact, when we started using this software, we realized that there are many good tutorials and manuals describing in great detail all the operations possible with the software - but we had a great deal of difficulty in finding and acquiring thematic maps, either because the tutorials are usually focused on a standard dataset (provided as an example of location data with GRASS) or because the information we needed was scattered all over the Net. It is our hope that the present document will contribute to filling this gap.

While waiting for our next article, readers seriously interested in the software can start by browsing the documentation pages available at the GRASS Web site, which also hosts some documents in non-English languages. We recommend the following documents:

Other resources available on the net can be useful:


Talkback: Discuss this article with The Answer Gang


[BIO]

I am a physicist involved with Linux since 1994, when I acted as system and security manager at the Department of Astronomy, University of Pisa (Italy). My current research interests are mainly in Applied Statistics, but I also work as system manager, net manager, web developer and programmer (C, C++, FORTRAN, Php, Tcl/Tk).



[BIO]

I am an astrophysicist mainly interested in galactic evolution; in this subject I develop simulation codes in C/C++ and FORTRAN. I've been using Linux since 1998, when I worked on my degree thesis at the University of Pisa (Italy).


Copyright © 2008, Matteo Dell'Omodarme and Giada Valle. Released under the Open Publication License unless otherwise noted in the body of the article. Linux Gazette is not produced, sponsored, or endorsed by its prior host, SSC, Inc.

Published in Issue 156 of Linux Gazette, November 2008

Writing Network Device Drivers for Linux

By Mohan Lal Jangir

Introduction

This article has been written for kernel newcomers interested in learning about network device drivers. It assumes that reader has a significant exposure to C and the Linux environment.

This article is based on a network driver for the RealTek 8139 network card. I chose the RealTek chip for two reasons: First, RealTek provides technical specifications for its chips free of cost. (Thanks, RealTek!) Second; it's quite cheap. It is possible to get the chip under Rs 300 (approximately US$7) in Indian markets.

The driver presented in this article is minimal; it simply sends and receives packets and maintains some statistics. For a full-fledged and professional-grade driver, please refer to the Linux source.

Preparing for Driver Development

Before starting driver development, we need to set up our system for it. This article was written and tested on Linux 2.4.18, which contains the source code for the RealTek8139 chip driver. It's very likely that the kernel you are running has the driver compiled either within the kernel itself or as a module. It's advisable to build a kernel which does not have the RealTek8139 driver in any form, to avert unnecessary surprises. If you don't know how to recompile the Linux kernel, I recommend you take a look at http://www.linuxheadquarters.com/howto/tuning/kernelreasons.shtml.

From this point of discussion onwards, it is assumed that you have a working kernel, which does not have driver for RealTek8139. You'll also need the technical specifications for the chip, which you can download from http://www.realtek.com.tw/. The last activity in this series is to properly insert the NIC into the PCI slot, and we are ready to go ahead.

It is strongly recommended to have Rubini's Linux Device Drivers book with you for quick API reference. This is the best resource known to me for Linux device driver development, as of now.

Starting Driver Development

Driver development breaks down into the following steps:

  1. Detecting the device
  2. Enabling the device
  3. Understanding the network device
  4. Bus-independent device access
  5. Understanding the PCI configuration space
  6. Initializing net_device
  7. Understanding RealTek8139's transmission mechanism
  8. Understanding RealTek8139's receiving mechanism
  9. Making the device ready to transmit packets
  10. Making the device ready to receive packets

Detecting the Device

As a first step, we need to detect the device of our interest. The Linux kernel provides a rich set of APIs to detect a device over the PCI bus (Plug & Play), but we will go for the simplest one and the API is pci_find_device.

#define REALTEK_VENDER_ID  0x10EC
#define REALTEK_DEVICE_ID   0x8139

#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/stddef.h>
#include <linux/pci.h>
int init_module(void)
{
    struct pci_dev *pdev;
    pdev = pci_find_device(REALTEK_VENDER_ID, REALTEK_DEVICE_ID, NULL);
    if(!pdev)
        printk("<1>Device not found\n");
    else
        printk("<1>Device found\n");
    return 0;
}

Table 1: Detecting the device

Each vendor has a unique ID assigned, and each vendor assigns a unique ID to a particular kind of device. The macros REALTEK_VENDER_ID and REALTEK_DEVICE_ID indicate those IDs. You can find these values from the "PCI Configuration Space Table" in the RealTek8139 specifications.

Enabling the Device

After detecting the device, we need to enable the device before starting any kind of interaction or communication with the device. The code snippet shown in Table 1 can be extended for device detection and enabling the device.

static struct pci_dev* probe_for_realtek8139(void) 
{
        struct pci_dev *pdev = NULL;
        /* Ensure we are not working on a non-PCI system *
        if(!pci_present( )) {
               LOG_MSG("<1>pci not present\n");
               return pdev;
        }

#define REALTEK_VENDER_ID  0x10EC
#define REALTEK_DEVICE_ID  0X8139

        /* Look for RealTek 8139 NIC */
        pdev = pci_find_device(REALTEK_VENDER_ID, REALTEK_DEVICE_ID, NULL);
        if(pdev) {
               /* device found, enable it */
               if(pci_enable_device(pdev)) {
                       LOG_MSG("Could not enable the device\n");
                       return NULL;
               }
               else
                       LOG_MSG("Device enabled\n");
        }
        else {
               LOG_MSG("device not found\n");
               return pdev;
        }
        return pdev;
}

int init_module(void) 
{
        struct pci_dev *pdev;
        pdev = probe_for_realtek8139();
        if(!pdev)
               return 0;

        return 0;
}

Table 2: Detecting and Enabling the Device

In Table 2, the function probe_for_realtek8139 performs the following tasks:

For time being, we temporarily suspend the thread of driver code study; instead, we look into some important topics in order to understand the Linux view of a network device. We will look at network devices, and the difference between memory-mapped I/O, port-mapped I/O, and PCI configuration space.

Understanding Network Devices

We have detected the PCI device and enabled it, but the networking stack in Linux sees interfaces as network devices. This is represented by the structure net_device. This means that the networking stack issues commands to the network device (represented by net_device), and the driver shall transfer those commands to the PCI device. Table 3 lists some important fields of the structure net_device, which will be used later in this article.

struct net_device 
{
  char *name;
  unsigned long base_addr;
  unsigned char addr_len;
  unsigned char dev_addr[MAX_ADDR_LEN];
  unsigned char broadcast[MAX_ADDR_LEN];
  unsigned short hard_header_len;
  unsigned char irq;
  int (*open) (struct net_device *dev);
  int (*stop) (struct net_device *dev);
  int (*hard_start_xmit) (struct sk_buff *skb,  struct net_device *dev);
  struct net_device_stats* (*get_stats)(struct net_device *dev);
  void *priv;
};

Table 3: Structure net_device

Although this structure has many more members, for our minimal driver, these members are good enough. The following section describes the structure members:

Although we have not mentioned all members of the net_device structure, please note especially that there is no member function for receiving packets. This is done by the device interrupt handler, as we will see later in this article.

Bus-Independent Device Access

Note: This section has been taken from Alan Cox's book Bus-Independent Device Accesses available at http://tali.admingilde.org/linux-docbook/deviceiobook.pdf

Linux provides an API set that abstracts performing I/O operations across all buses and devices, allowing device drivers to be written independent of bus type.

Memory-Mapped I/O

The most widely supported form of I/O is memory-mapped I/O. That is, a part of the CPU's address space is interpreted not as accesses to memory, but as accesses to a device. Some architectures define devices to be at a fixed address, but most have some method of discovering devices. The PCI bus walk is a good example of such a scheme. This document does not cover how to receive such an address, but assumes you are starting with one.

Physical addresses are of type unsigned long. These addresses should not be used directly. Instead, to get an address suitable for passing to the functions described below, you should call ioremap. An address suitable for accessing the device will be returned to you.

After you've finished using the device (say, in your module's exit routine), call iounmap in order to return the address space to the kernel. Most architectures allocate new address space each time you call ioremap, and they can run out unless you call iounmap.

Accessing the device

The part of the interface most used by drivers is reading and writing memory-mapped registers on the device. Linux provides interfaces to read and write 8-bit, 16-bit, 32-bit and 64-bit quantities. Due to a historical accident, these are named byte, word, long, and quad accesses. Both read and write accesses are supported; there is no prefetch support at this time. The functions are named readb, readw, readl, readq, writeb, writew, writel, and writeq.

Some devices (such as framebuffers) would like to use larger transfers that are more than 8 bytes at a time. For these devices, the memcpy_toio, memcpy_fromio and memset_io functions are provided. Do not use memset or memcpy on I/O addresses; they are not guaranteed to copy data in order.

The read and write functions are defined to be ordered. That is, the compiler the the the is not permitted to reorder the I/O sequence. When the ordering can be compiler optimized, you can use __readb and friends to indicate the relaxed ordering. Use this with care. The rmb provides a read memory barrier. The wmb provides a write memory barrier.

While the basic functions are defined to be synchronous with respect to each other and ordered with respect to each other the buses the devices sit on may themselves have asynchronocity. In particular many authors are not comfortable by the fact that PCI bus writes are posted asynchronously. An author of the driver must issue a read from the same device to ensure that writes have occurred in the manner the author wanted it. This kind of property cannot be hidden from driver writers in the API.

Port Space Access

Another form of I/O commonly supported is Port Space. This is a range of addresses different from the normal memory address space. Access to these addresses is generally not as fast as accesses to the memory mapped addresses, and it also has a potentially smaller address space.

Unlike with memory mapped I/O, no preparation is required to access port space.

Accessing Port Space or I/O mapped devices

Accesses to this space are provided through a set of functions which allow 8-bit, 16-bit and 32-bit accesses; also known as byte, word and long. These functions are inb, inw, inl, outb, outw and outl.

Some variants are provided for these functions. Some devices require that accesses to their ports are slowed down. This functionality is provided by appending a _p to the end of the function. There are also equivalents to memcpy. The ins and outs functions copy bytes, words or longs to/from the given port.

Understanding PCI Configuration Space

In this section, we will look at PCI configuration space. PCI devices feature a 256-byte address space. The first 64 bytes are standardized while the rest of the bytes are device dependent. Figure 1 shows the standard PCI configuration space.

PCI Configuration Space

Figure 1: PCI Configuration Space

The fields "Vendor ID" and "Device ID" are unique identifiers assigned to the vendor and the device, respectively. (We have seen them in the section "Device Detection".) Another field to note is "Base Address Registers", popularly known as BAR. We will see how BARs are used shortly.

Initializing net_device

Now it's time to revert back to driver development. Before that I remind you about the priv field of the structure net_device. We will declare a structure which holds data private to our device and that structure shall be pointed to by member priv. The structure has the following members (We will update structure members as we progress).

struct rtl8139_private 
{
        struct pci_dev *pci_dev;  /* PCI device */
        void *mmio_addr; /* memory mapped I/O addr */
        unsigned long regs_len; /* length of I/O or MMI/O region */
};
           

Table 4: rtl8139_private structure

Now we define a net_device pointer and initialize it, in the rest of the init_module function.

#define DRIVER "rtl8139"
static struct net_device *rtl8139_dev;

static int rtl8139_init(struct pci_dev *pdev, struct net_device **dev_out) 
{
        struct net_device *dev;
        struct rtl8139_private *tp;

        /* 
         * alloc_etherdev allocates memory for dev and dev->priv.
         * dev->priv shall have sizeof(struct rtl8139_private) memory
         * allocated.
         */
        dev = alloc_etherdev(sizeof(struct rtl8139_private));
        if(!dev) {
               LOG_MSG("Could not allocate etherdev\n");
               return -1;
        }

        tp = dev->priv;
        tp->pci_dev = pdev;
        *dev_out = dev;

        return 0;
}

int init_module(void) 
{
        struct pci_dev *pdev;
        unsigned long mmio_start, mmio_end, mmio_len, mmio_flags;
        void *ioaddr;
        struct rtl8139_private *tp;
        int i;

        pdev = probe_for_realtek8139( );
        if(!pdev)
               return 0;

        if(rtl8139_init(pdev, &rtl8139_dev)) {
               LOG_MSG("Could not initialize device\n");
               return 0;
        }

        tp = rtl8139_dev->priv; /* rtl8139 private information */
        
        /* get PCI memory mapped I/O space base address from BAR1 */
        mmio_start = pci_resource_start(pdev, 1);
        mmio_end = pci_resource_end(pdev, 1);
        mmio_len = pci_resource_len(pdev, 1);
        mmio_flags = pci_resource_flags(pdev, 1);

        /* make sure above region is MMI/O */
        if(!(mmio_flags & I/ORESOURCE_MEM)) {
               LOG_MSG("region not MMI/O region\n");
               goto cleanup1;
        }
        
        /* get PCI memory space */
        if(pci_request_regions(pdev, DRIVER)) {
               LOG_MSG("Could not get PCI region\n");
               goto cleanup1;
        }

        pci_set_master(pdev);

        /* ioremap MMI/O region */
        ioaddr = ioremap(mmio_start, mmio_len);
        if(!ioaddr) {
               LOG_MSG("Could not ioremap\n");
               goto cleanup2;
        }

        rtl8139_dev->base_addr = (long)ioaddr;
        tp->mmio_addr = ioaddr;
        tp->regs_len = mmio_len;

        /* UPDATE NET_DEVICE */

        for(i = 0; i < 6; i++) {  /* Hardware Address */
               rtl8139_dev->dev_addr[i] = readb(rtl8139_dev->base_addr+i);
               rtl8139_dev->broadcast[i] = 0xff;
        }
        rtl8139_dev->hard_header_len = 14;

        memcpy(rtl8139_dev->name, DRIVER, sizeof(DRIVER)); /* Device Name */
        rtl8139_dev->irq = pdev->irq;  /* Interrupt Number */
        rtl8139_dev->open = rtl8139_open;
        rtl8139_dev->stop = rtl8139_stop;
        rtl8139_dev->hard_start_xmit = rtl8139_start_xmit;
        rtl8139_dev->get_stats = rtl8139_get_stats;

        /* register the device */
        if(register_netdev(rtl8139_dev)) {
               LOG_MSG("Could not register netdevice\n");
               goto cleanup0;
        }

        return 0;
}
               

Table 5: net_device initialization

It's time to explain what we have done in Table 5. Function probe_for_realtek8139, we have already seen. Function rtl8139_init allocates memory for global pointer rtl8139_dev, which we shall be using as net_device. Additionally, this function sets the member pci_dev of rtl8139_private to the detected device.

Our next objective is to get the base_addr field of the net_device. This is the starting memory location of device registers. This driver has been written for memory-mapped I/O only. To get the memory-mapped I/O base address, we use PCI APIs like pci_resource_start, pci_resource_end, pci_resource_len, pci_resource_flags etc. These APIs let us read the PCI configuration space without knowing internal details. The second argument to these APIs is the BAR number. If you see, RealTek8139 specifications, you will find that the first BAR (numbered as 0) is I/OAR, while second BAR (numbered as 1) is MEMAR. Since this driver is using memory-mapped I/O, we pass the second argument as 1. Before accessing the addresses returned by the above APIs, we have to do two things. First is to reserve the above resources (memory space) by driver; this is done by calling the function pci_request_regions. The second thing is to remap I/O addresses as explained in section above on Memory-Mapped I/O. The remapped io_addr is assigned to the base_addr member of the net_device, and this is the point where we can start to read/write the device registers.

The rest of the code in Table 5 does straightforward initialization of net_device. Note that now we are reading the hardware address from the device and assigning it to dev_addr. If you see "Register Descriptions" in RealTek8139 specification, the first 6 bytes are the hardware address of the device. Also we have initialized function pointer members but haven't defined any corresponding function. For time being, we define dummy functions to compile the module.

static int rtl8139_open(struct net_device *dev) { LOG_MSG("rtl8139_open is
called\n"); return 0; }

static int rtl8139_stop(struct net_device *dev) 
{
        LOG_MSG("rtl8139_open is called\n");
        return 0;
}

static int rtl8139_start_xmit(struct sk_buff *skb, struct net_device *dev) 
{
        LOG_MSG("rtl8139_start_xmit is called\n");
        return 0;
}

static struct net_device_stats* rtl