GAUSSIAN FIT IN PYTHON

Gaussian fitting in Python… what a pain. Have you ever wanted to fit some data to a Gaussian in Python in 2 minutes? I did. It’s easy right… there must be a function that does least square fits … hopefully there’s some instructions on how to use it. Call it with a Gaussian as the work function and you’re done. Or so I thought. I also thought it would be smart to get on stackoverflow.com and see what code lurked around there. So this is the story of what I found:

I started with this incorrect – just plain wrong version (note to self 1: post a message correcting it). Then I got to this version which is confusing if you want to get a Gaussian in 2 minutes. Though this last version had an interesting side link – which even though not about Gaussians is somewhat relevant to fitting functions. Finally I get to this correct starting point – which was a non-fitted version of what I wanted (note to self 2: in the future use the Cookbook more than stackoverflow). Then once I got confident I was on the right path there’s this bug related to using floats – a strange apparition waiting to ruin my day.

So without much further ado here is – finally – my correct and functional version:

#defines model function for the data to be fitted to
def gauss(x, *p):
   A, mu, sigma = p
   # I need to cast denominator and numerator as float64 because of a 
   # bug in numpy
   numer = np.float64(-(x-mu)**2.)
   denom = np.float64((2.*sigma**2.))
   retVal = A*np.exp(numer/denom)
   return retVal

def fitToGaussian(xData, yData):
   A = yData.max()
   mean = (sum(xData*yData)/sum(yData))
   sigma =((sum(yData*(xData-mean)**2)/sum(yData)))
   print ('A:' + str(A) + '; mean: ' + str(mean) + 
          '; sigma: ' + str(sigma))
   # p0 is the initial guess for the fitting coefficients (A, mu and 
   # sigma)
   p0 = [A, mean, sigma]
   coeff, var_matrix = curve_fit(gauss, xData, yData, p0=p0)
   return coeff

dataX = loadDataX()
dataY = loadDataY()

coeff = fitToGaussian(dataX, dataY)

MAKING WINDOWS7 IMAGES

So you just finished setting up your Windows7 machine just the way you like it. Now that all is good you don’t want to waste another 4 hours next time you want to reinstall everything from scratch. Or maybe you are part of a small start-up where you are the one and only software developer / IT department and would like to replicate this same installation on computers that accompany devices you are manufacturing and selling to customers. Don’t despair this is a guide that uses Microsoft’s WindowsPE (Preinstallation Environment) to satisfy your desires.

Before you start you will need:

  • Item1: Windows 7 installation disk. These are increasingly harder to come by.
  • Item2: 1x WindowsPE bootable USB memory stick (at least 16GiB). See below on how to set up WindowsPE.
  • Item3: 1x empty bootable USB memory stick (at least 16GiB) which will become the back-up “disk”.

NOTE: Make sure both USB memory sticks are formatted with NTFS file system. If they are not and they are formatted with FAT32 FS the maximum file size cannot exceed 4GiB. A Windows install image file is around 4GiB-10GiB so the imaging process will fail.

Backup process:

Ensure that the Windows7 machine has been configured to your satisfaction (i.e. driver and programs of interest installed, bloatware removed, default parameters and settings set etc) then restart the computer and boot off the Windows PE USB stick (Item2).

Once WindowsPE is running execute:

g:\imagex.exe /compress fast /capture d: g:\install.wim "Windows7"

where g:\ drive is the drive assigned to the WindowsPE USB drive and d:\ drive is the drive assigned to the Windows7 installation you want to save.

NOTE: You might need to run a few dir commands to figure out which is which. I usually just try dir c:\, dir d:\ up to dir g:\ and from the directory listing deduce which drive I want to back-up.

The image creation takes about 30-60 minutes depending on how large the installed software on the Windows7 drive is, so be patient. At the end of it, on the WindowsPE USB drive (Item2), you will have a file called install.wim. We will need this a little later.

Next we take the empty USB memory stick (Item3) and make it into a bootable USB device by following the description below. Then we copy all the files of the Windows7 installation disk (Item1) onto it (Item3). Next you will replace the default .wim file found in the sources directory of Item3 with the recently created install.wim file.

When all this is said and done this last USB stick (Item3) will contain an installation kit of Windows7 that if deployed will have all your settings pre-configured. One caveat is that this Windows7 back-up can only be installed on the IDENTICAL hardware it was made on. This doesn’t mean the exact same machine but it does mean the exact same model of computer – all the devices installed on the initial machine need to be present on the machine on which the back-up/restore installation kit will be set-up. If one of the controllers (i.e. a different type of HDD) is different the installation will fail miserably.

How to make a bootable USB memory stick in Windows7:

Insert an appropriately sized (16GB at least) memory stick in a USB port and then run the following commands from a command prompt console (running with Administrator privileges):

diskpart
list disk
select disk 1

[make sure disk 1 is the USB memory stick disk – use the size of the disk as indicator of which disk to select. If you choose wrong you will obliterate your local OS installation]

clean
create partition primary
select partition 1
active
format fs=ntfs

 [we don’t want to format as FAT32 because maximum file-size for FAT32 partitions is 4GiB. In our case the instal.wim file will most likely be larger than that]

assign
exit

At the end of this process you will have a bootable USB memory stick.

How to make a WindowsPE bootable USB memory stick:

Follow the steps above on how to make a bootable USB memory stick. Then copy the files in this archive on it (unarchive the files from the .zip file before copying them over) and you will have a WindowsPE bootable USB memory stick.

If you would like to make your own version of WindowsPE go ahead and follow the instructions on the Microsoft website. Oh yeah now that you are going full bore you will probably want to download Microsoft AIK tools too. Good luck!

After installing the backup image:

Once a backup image is installed you will need to change the product key and authenticate the new installation.

To change the product key start a cmd.exe as Administrator and run:

slmgr.vbs -ipk [PRODUCT-KEY]

After about 1 minute the OS will display a pop-up informing you the product key has been successfully changed.

To authenticate use the same command prompt and run:

slmgr.vbs -ato

That’s all.

START-UP IT ENVIRONMENT

Let’s say you just finished your undergrad in computer-science and got hired as the lead developer in a small start-up that’s developing a novel hardware product. How would you go about choosing the tools of your trade?

Here is my high-level take on a few of the nodes in a software group structure and the information flow between them. I am not going to discuss any tool-chains in this post as they are slightly lower level than the scope I want to present.

NOTE 1: The following configuration is derived from my experience developing code for an architecture that contains some hardware driven by a micro-controller which then connects to a computer running an application that is the interface to the user.

NOTE 2: This IT infrastructure is most likely NOT IDEAL for web-centric app development.

IT_Map

IT Structure

I will now go through (in alphabetical order) a few of the nodes and explain why I think they are important:

Box1 – Code Server:

The Code Server stores all the code related files. In my case it runs a Mercurial server for source control management and a Redmine server for bug tracking/feature requests and general software project maintenance. I chose Mercurial (as opposed to the dominant Git) because it has a high available-functionality to complexity ratio. A few of the features that made me stick with Redmine are that it integrates nicely with Mercurial, it is fairly customizable using a range of plugins and it is free. One of the downsides of using Redmine is that it’s a little trickier to update.

As far as the actual platform, the Code Server, is a VirtualBox running off Ubuntu OS. I assign it initially 64GiB hard drive space with 2GiB of RAM. As the software project grows you might need to assign it more space.

Box2 – Wiki Server:

A wiki is one of the center piece software modules of a start-up. It can store anything from contact info for suppliers and purchasing procedures to assembly instructions and recycling locations. The principle that anyone can edit it and improve it creates a hub for all members of the team to congregate around. For the wiki engine I choose MediaWiki over DokuWiki. DokuWiki is great because it does not need a mysql DB, however it does not have as an extensive set of plugins as MediaWiki.

Hardware wise, this is also a VirtualBox running Ubuntu. 32GiB hard drive space and 1GiB RAM should suffice. I like to keep the Wiki Server decoupled from the Code Server because as it has a higher user demand ideally it has as little downtime as possible. The fewer services the Wiki Server runs the more likely it is that it won’t need a restart.

Box3 – NAS:

This is real hardware and as the company file server is one of the corner stone of the Software Group. As I don’t like to maintain file servers I default to a Synology’s DS412 Disk Station NAS. Even though they it is a little more expensive (as of 2013 it is $700 and with 4x 3TiB $600 it comes in at $1300 all in) the piece of mind of not scrambling to fix someone’s file access problems, when your plate is full with fixing high priority bugs and implementing some last feature that changes 50% of the software architecture, more than makes up for it. The Synology box is your file storage solution and as such you wouldn’t want anything bad to happen to it. What I am driving at is that all the data on it needs to be backed up somewhere safe. When it comes to back-up solutions I like to use CrashPlan. They have great client software and good prices. Not to mention I like they’re corporate attitude – just go and read some of their job postings and you’ll know what I mean, young paduan.

Host Box:

This is the real hardware PC on which all the virtual boxes will run. As such it needs to be adequately fast and stable. As of this writing (2013) I would like the following configuration an i7 3770k CPU ($350) with 16GiB of RAM ($150) and a 256GiB SSD ($200). No need for a monitor or a stand alone graphics card as this box won’t be running anything fancy. Though you will need a motherboard ($150), a case w/ power supply ($150) and a puny keyboard ($10).  I like to think that running a Windows7 64bit Professional  ($150) as the host OS is slightly better than a Linux distor just because of the drivers support available. I seem to think Microsoft and Intel optimize their packages for each other but I admit I didn’t do any research into it. All of the above comes in at around ($1200) and will form the other corner stones of the Software Group.

  • i7 3770k CPU (NCIX sku: 70540)
  • 16GiB DDR3 RAM (NCIX sku: 66354)
  • 256GiB SSD (NCIX sku: 60446)
  • Motherboard (NCIX sku: 71114)
  • PC Case (NCIX case sku: 55583
  • Ppower supply (NCIX sku: 67325)
  • Keyboard (NCIX sku: 54063)

Virtual Box:

I happen to think that Virtual Box is one of the best creations in OS history. It allows for decoupling of the OS from the underlying hardware and more than that it provides for a straight forward, fast and robust backup and restoring procedure. I can export my VirtualBox client which contains all the source control and issue tracking repositories I develop on, move it on a USB stick and then import it on my laptop’s VirtualBox. This way I can have my server with me wherever I go. Yes, yes it’s actually not as simple as this and with great power comes great responsibility but these are topics for another discussion.

Crash Plan:

I have been using Crash Plan as my cloud backup solutions for a few years now. It has a simple interface and allows for incremental backups. The data is encrypted locally and then uploaded into the cloud which means Crash Plan doesn’t have access to my files. For a small start-up needs I find it more than adequate. A few caveats related to backing up. The original seed backup will take a long time. If the full backup needs to be restored expect a few days to get it on HDDs. However in this case I find that in most cases only a small percentage of the backed up data is touched during daily operations. Most of the backed up data being stored on the NAS being stale. Ideally the start-up has a fast connection (most often the case if it’s a spin-off from an university lab research) and in case of NAS failure a small daily-work chunk of data can be immediately restored and the users can continue working off it.


 

Having a solid infrastructure for the software team from the get-go allows you to focus more on the development of the core software rather than on development of maintenance procedures and ancillary software.