When dealing with processing really large files (300files~15MBeach), it quickly becomes a process that you have to wait a long time for it to finish. I wrote a Java program that manipulated a flat text P1/90 (Seismic Vessel’s Positions) file and outputed a corrected version in another directory. Thus I wanted to check how far along it had gone and when I should expect it to finish.

Mind you I could have done this in the Java code – keep a record of all the files and how many had been processed – but this project was time-constrained so I didn’t have the luxury of developing it the way I wanted; it had to be done quickly. I therefore delegated the monitoring to the command line tools. Since I new how many files I had originally and how many were being created in the new directory, all I had to do is count each of them and compare them. The following one-liner does exactly that:

watch 'ls *190 | echo -n "`wc -l`"; echo -n "/"; ls ../*90 | wc -l'

The watch command executes a command periodically and prints the output on the terminal. It could be parametrised further by defining the time interval of execution with the -n flag, e.g. watch -n 60 would execute the command every 60 seconds.  The above watch command is enclosed in single quotes and does the following:

• ls *90 | echo -n “`wc -l`” : This will enumerate the files ending in “90″ in the current directory (the one I was creating the output files) and pipe that into a print command. The print command is the echo. The -n flag removes the newline character at the end, so that we can keep the printout in the same line. The print output of the echo is the word count (wc) of how many filenames the ls command return. This gives us the total number of files found in this directory.
• echo -n “/” : This will append in the same line a forward slash. This is also inline with the previous output and creates the appearance of a ratio.
• ls ../*90 | wc -l :The final command will print, with a newline at the end, the number of files in the original – above – directory.

The output of this command looked like this:

Every 2.0s: ls *190 | echo -n "`wc -l`"; echo -n "/"; ls ../*90 | ... Tue Apr 10 16:51:00 2012
288/336

Do you have similar examples you can contribute?

I’ve decided to set a challenge for myself. It’s been a while since I actively did any kind of training/exercise. Partly because of feeling some anxiety from a recent hernia operation, I was hesitant to exert myself. Not anymore!

I was looking for something to push me into completing a challenge and what better way to do this than doing it to raise money for a charity. When I saw the Swimathon Weekend 2012 I knew I had to sing-up.

So this is me committing to swim 5 Km for the Marie Curie Cancer Care.

Being a true analytics guy, I did the only logical thing I could think of. I began to track my progress to make sure I’ll be able to complete the 5 Km. At this location you can, if you so wish, see my progress until the Swimathon on April 27th. I’ll try and test completing a 5K swim before that date, just to be sure I can do it. If you wan to show your support in any way you can head over to my Swimathon appeal page. There you can either send me a supportive message, or better yet make a donation.

The donation will help the Marie Curie Cancer Care nurses, help take care people with cancer in their own home. Please give as much as you can. Even £5 is enough if it comes from 50 people!

I plan to train until I’m sure I can do the 5K for this challenge. This will require some determination and endurance on my part, having to juggle work, my MSc, and the training, but I’m really excited to do this. I hope you can help support me any way you can.

I appreciate everyone’s support in whatever form it may come.

Thank you,

Christos

Here’s an excerpt:

A San Francisco cable car holds 60 people. This blog was viewed about 2,300 times in 2011. If it were a cable car, it would take about 38 trips to carry that many people.

Click here to see the complete report.

As I wrote some time ago, I recently migrated from linux to OSX. Since this OS was very new to me, it took me a while to figure out which applications were useful to me and which customisations added to my productivity. This article is a list for people new to OSX and want to hit the ground running. It is basically the article I would have liked to have found when I first migrated.

Frist off let me start with the applications I’ve found to be the most useful and then pass on to the productivity customisations I’ve attempted.

The apps that are absolutely essential to me are:

• Google Chrome, for the plugins, but Safari is a close second for the zoom function when double tapping.
• Spotify, for the ease of finding music and playing it anywhere with a Premium account.
• Skype, essential tele-comms.
• NetNewsWire, RSS reader, ad-supported.
• Arq, off-site (AWS) backups.
• iTerm2
• Parallels, for some simulation programs for uni that only run on windows.
• SofaControl + Plex. The ideal media player setup.
• 1password + Dropbox: Always set up secure long passwords, access them from all the machines, and only remember one password.
• Air Video Server: Play any movie on the iPad, that’s stored on the iMac. All videos are streamed over WiFi.
• Caffeine + Flux: Two ways to control the luminosity of the screen. 50% Brightness and 3200K after dark is ideal for me.
• Evernote +MindNode: Notes and ideas.
• Handbrake + VLC: video conversion and playback.
• iBank: Finances’ handling
• Remoteless helper: To control Spotify on the iMac from the ipad.
• DaisyDisk: Disk Utilisation
• SpiderOak/Dropbox: File sharing. SpiderOak is much safer. You can use your own git server and the encryption keys stay with you not the company.
• SizeUp: Window position management
• TeamViewer: Fix my mom’s PC from a distance!
• LatTeXiT/TeXShop: LaTeX equation editor and editor with PDF preview.
• TweetDeck: Multiple account plus Unicode.
• Sophos Anti-virus: It is kind of a myth that Macs don’tneed antivirus.
• Seahorse/GIMP: Simple image manipulation.

A special mention to this list must be GeekTool with regards to customisation. This tool has allowed me to set up and show quite a few useful information on my desktop. You can see the screengrab below for an example.

This tool adds weather info, iCal appointments, TODOtxt list, date/time, and activity status all on the desktop. Quite easily customisable as well via scripts.

The other powerful feature that exists within OSX is of course the command line. It’s not a complete UNIX environment but with the help of Homebrew most of the UNIX tools can be compiled and installed easily.

By running:

brew install aspell --lang=en-GB

you get a spell-checker that is ideal for correcting LaTeX files:

aspell -c --mode tex filename

I install most brew recipes with:

brew install spark todo-txt agrep dos2unix fortune gawk git gnu-sed htop multitail psgrep rename sqlite ssh-copy-id wget

You can get more brewing! recipes here.

The rest of the work I do is on the command line. For this I use iTerm2 which has to be the terminal OSX should’ve had for a long time. Multiple tabs and easy customisation make it a treat to work with.

I use the Solarized color theme for both iTerm2 and ViM. You can find both here. I find it to be the most eye-pleasing theme and also the most legible. There is both a dark and light theme depending on your preference. With some scripting you could even have the light theme during the morning (with natural light) and switch to the dark theme at night.

I’m come back to ViM customisation, but not before I mention a really cool customization for the zsh. I’ve been using zsh for a while now (In greek only, sorry). I’ve spent some time trying to customise my Prompt to show relevant information but also look pleasant to the eye. I think oh-my-zsh achieves both and then more. Functions, autocompletes, aliases and ready made themes. You can see some screenshots of various themes here. You can install it quite easily like so:

wget --no-check-certificate <a href="https://github.com/robbyrussell/oh-my-zsh/raw/master/tools/install.sh">https://github.com/robbyrussell/oh-my-zsh/raw/master/tools/install.sh</a> -O - | sh

or by cloning the git repo:

git clone git://github.com/robbyrussell/oh-my-zsh.git ~/.oh-my-zsh

For me I’ve settled to the fino theme. You can see a screenshot below.

At the prompt, aside from user and hostname info, you also get PWD and the git brach and status of the directory you are.

As for the rest of the terminal configuration I want to mention the ViM customisations since I’ve already said some things on GNU Screen (Again apologies, this is in Greek. I will try and translate them into English if there’s any interest.).

My ViM config file emphasises on functionality. I like having line numbers visible at all times, and I’ve also deactivated the arrows navigation in order to force me to use the h,j,k,l navigation keys. I’m also using Pathogen in order to load modules like the TagList, TaskList, NERDTree, and the solarized color theme of course. You can see the end result below.

At the bottom of the screen what you’re seeing is the PowerLine plugin. A very nifty tool that provides immediate information about the file you’re editing. There’s also a visual cue whether you’re in (n)ormal mode or (i)nsert mode.

Normal Mode

Insert Mode

I’m also in the process of turning my ViM setup into a complete Python IDE, but unfortunately I’m having som estrange issues compiling ViM from source on OSX with python support. Any thoughts as to how I can do this? I have followed advice so far from these really good sources for this setup.

The only other aspect I wanted to have was a way to sync these rc config files between my iMac and my MBP. I wanted the customisations made on one machine to be automatically reflected on the other. The way I did this, although you could do the same with Dropbox [disclaimer: this is a referral link, you can open a new browsing tab and just type the URL], is via iCloud. I saw somewhere online that if two Macs are syncing to the same iCloud account, as mine are, then the directory ~/Library/Mobile\ Documents/ is automatically synced between the machines with the same account. so what I did is placed all my rcfiles and .vim directory in that location and soft linked to them from home, e.g.:

UPDATE: It appears the above method is not very reliable. Having done this to sync the iMac with the MacBook, without any warning (which is reasonable since this isn’t really mainstream use of the location), the contents of the Mobile Documents directory were erased. Therefore I have reverted to the same setup as below, with symlinks, but via my Dropbox path. I am in the process of setting up a SparkleShare server, but that’s a different post at a later time.

Soft Links

This will create a link to the sync’ed version of vimrc which can be accessed by your local ViM installation.

I hope you’ve found this article useful in some way. Let me know in the comments if you have a cool/useful customisation you’ve done that you’d like to share.

As usual you can find all the config files used for this setup in my GitHub repo. Fork away!

P.S. I should also mention that the font seen in these screengrabs is the Open Font – Anonymous Pro. You can get it here.

In a previous post I presented an approach to solve the shortest path problem using different implementations of algorithms. The shortest path problem is a classic and important combinatorial optimisation problem. It often appears as a subproblem when solving difficult combinatorial problems like the multicommodity network flow (MCNF) problems.

The problem is to route a set of demands (commodities) through a network¹ with limited arc capacities. The inherent assumption with the following implementation is that the commodities can be shipped in fractions, hence are divisible.

The technique that is implemented in the Java code is the Column Generation a.k.a. the Dantzig-Wolfe Decomposition. This technique has many applications to practical problems. For instance routing data packets through telephone networks, routing electricity demand in power grids, routing vehicles on the road network, allocating airline crews to flights, production scheduling and planning, etc.

The column generation algorithm can be interpreted as applying the Simplex Method to a Linear Problem with a huge number of columns. Only a few columns are present at the start, other columns are generated when needed.

Problem Example

Taking the small network shown below as an example, we can state that all edges are bidirectional with a cost of 1 and a total capacity of 2.5 (D-F has 2.6)

If the commodities:

• A-F: 3 units
• e-D: 2 units

Implementation

References

1. The network in the present example is defined as a graph V, which is a set of vertices V= {1,…n}.

It also includes a set j in E of (directed) edges specified by a start and end verted and also the cost and cpacity of each vertex: Cj, CAPj.

Finally a set i in C of commodities which is given by the source and target of each commodity: Si, Ti, and the demand Di.

2. You can find the library at http://lpsolve.sourceforge.net. To use it on Windows you need to copy the shared libraries lpsolve55.dll, lpsolve55j.dll as well as the interface lpsolve55ja.jar into the project workspace.

To compile use:

javac -cp lpsolve55ja.jar;. Mcnf.java

To run the program use:

java -cp lpsolve55ja.jar;. Mcnf

Remark

For OSX you’ll needto compile from source. Change to the lib/mac directory and edit the file build_osx. Change the directory paths as indicated in the comments. Follow the compilation instructions to get the binaries of the solver

Dynamic programming is a useful mathematical technique for making a sequence of inter-related decisions. It provides a systematic procedure for determining the optimal combination of decisions, given the constraints at hand. In contrast to linear programming, there is no standard formulation for the mathematical problem. Hence each problem requires analysis and modelling for itself. A large number of algorithms are also available for this field of problems.(1)
The example above, taken from Hillier & Lieberman’s “Introduction to Operational Research”, describe the Program Evaluation and Review Technique (2) the interdependent tasks A, B, C, D, and E. Each task has an estimated time of completion -3, 4, 5, 3, and 4 respectively-. Unless process C finishes, task E cannot begin to execute.
This process is commonly used in project management scenarios, when inter-related events need to have their predecessors completed before taking up a certain task.

The purpose of this scenario is to find the expected time of completion for this project. The uncertainty comes from the fact that each of the times of completion are estimations. Therefore the scenario needs to be run multiple times and then the expected completion time to be calculated. The distribution of the completion times will be outputted as the result.

There are three milestones to this project:

1. Parse an ASCII data file and construct a Graph to represent the problem.
2. Find the critical path through the graph.
3. Calculate the experimental distribution of the total project duration, given distributions for the completion time of each sub-task.

A. Task 1
——————-

The files will have the following syntactic and semantic structure:

0        0        0        0
A        1        2        3        0
B        2        3.5        8        0        A
C        6        9        18        0        B
D        4        5.5        10        0        C
E        1        4.5        5        0        C
F        4        4        10        0        E
G        5        6.5        11        0        D
H        5        8        17        0        E        G
I         3        7.5        9        0        C
J        3        9        9        0        F        I
K       4        4        4        0        J
L        1        5.5        7        0        J
M       1        2        3        0        H
N        5        5.5        9        0        K        L
Z        0        0        0        A B C D E F G H I J K L M N

The first column is the [name] of the milestone.
The second column is the [opt]imistic estimate of duration.
The third column is the most [likely] estimate of duration.
The fourth column is the [pess]imistic estimate of duration.
From then on the remaining characters indicate preceding milestones that this milestone [depends] on.

Once all the lines have been read, they’ll be used to construct the graph of the project. This graph will use the methods in the Graph class of the code.
From the time of completion selection a random variate will be selected based on the likely, best and worst case scenarios.

The duration of each activity will be a Normally Distributed Random Variate (NDRV) with mean μ=(o+4m+p)/6 and variance σ^2=(p-o)2/36, where o, m, p the given optimistic, likely, and pessimistic times of completion.
JAVA’s Random class was used to generate Normally distributed random numbers N(0,1). This value was then multiplied by the variance calculated above for the times of completion and subsequently the newly calculated mean was added to this value to produce an NDRV(μ, σ^2).

B. Task 2
——————-

For the critical path method, different algorithms were tried. Dijkstra’s algorithm had to be modified to work for the critical path. Since Dijkstra’s algorithm works efficiently to find the minimum cost path, the time of completions had to be inverted in order to look for the lowest inverted time. This didn’t produce optimal solution, plus had additional problems with potential 0 time of completion or negative times.
The Backwards propagation algorithm could not handle a terminal node with all edges having 0 time of completion and thus couldn’t start at all.
The forwards propagation algorithm couldn’t handle large costs deeper in the graph and thus did not produce an optimal solution. It did however produce feasible solutions.
The algorithm finally utilised was the Backwards propagating Bellman-Ford(BF) algorithm. The Bellman–Ford algorithm computes single-source shortest paths in a weighted digraph. Bellman–Ford runs in O(|V|·|E|) time, where |V| and |E| are the number of vertices and edges respectively. (3) An added benefit of the BF algorithm is that it can identify negative cycles and break the execution.

C. Task 3
——————-

The mean, variance, and distribution of the simulation will be outputted on the terminal and a small GUI showing the histogram. Normally to calculate the variance of the data set, two passes of the data would be required. One to calculate the expected value of the series and a second to calculate the residuals of the values to the expected value.
var = Sum(Xi-Xmu)^2)/N , for all i belongs to N

However by decomposing the above formula, it is possible to do this with one run, to minimise the time of execution:
var = (Sum(Xi))^2/N – (Sum/N)^2 , for all i belongs to N

The bin size is set at 1 for the min-max range of (30,59), although by calculating the range of the series the following formula can be used to determine the optimal bin size (W):

W=3.49xSDxN^(-1/3), where SD is the sample standard deviation and N the available data point. (4)

Conclusion
——————-
The final outcome indicates that for the case102.dat (included in the repo) the statistical output of the simulation and the logarithmic histogram are:

Experiment's mean value for the critical length paths: 43.41049912306386
Experiment's variance value for the critical length paths: 12.700112769478892
Critical Path: 0 - E - O - P - Q - Z
Bin        Prob        LogHist                Count
30        0%               |                0/10000
31        0%               |                0/10000
32        0%               |@                5/10000
33        0.2%             |@@@                22/10000
34        0.2%             |@@@                25/10000
35        0.8%             |@@@@        83/10000
36        1.6%             |@@@@@        155/10000
37        2.8%             |@@@@@        279/10000
38        4.4%             |@@@@@@        440/10000
39        6.3%             |@@@@@@        626/10000
40        8.4%             |@@@@@@        840/10000
41        9.8%             |@@@@@@        983/10000
42        11.6%            |@@@@@@@        1164/10000
43        11.2%            |@@@@@@@        1120/10000
44        10.3%            |@@@@@@        1028/10000
45        9.2%             |@@@@@@        920/10000
46        7.7%             |@@@@@@        774/10000
47        5.7%             |@@@@@@        570/10000
48        3.8%             |@@@@@        375/10000
49        2.6%             |@@@@@        257/10000
50        1.6%             |@@@@@        162/10000
51        0.9%             |@@@@        93/10000
52        0.4%             |@@@                44/10000
53        0.2%             |@@@                22/10000
54        0.1%             |@                6/10000
55        0%               |@                3/10000
56        0%               |                1/10000
57        0%               |                0/10000
58        0%               |                0/10000
59        0%               |                0/10000

This histogram exhibits the normal distribution expected from the simulation with the mean and distribution provided by the o, m , p times of the milestones.

This project was awarded 72/100 marks.

As usual the source for this project can be checked out and forked at https://github.com/cmdel/javaPERT.

REFERENCES
============

1. http://en.wikipedia.org/wiki/Dynamic_programming#Algorithms_that_use_dynamic_programming
2. http://en.wikipedia.org/wiki/Program_Evaluation_and_Review_Technique
3. http://en.wikipedia.org/wiki/Bellman%E2%80%93Ford_algorithm
4. Scott, 1979. On optimal and data-based histograms. Biometrika, 66:605-610.

As part of the MSc course I’m currently undertaking, there’s a great need to understand optimality of processes and operations. One of them is the problem of queues while serving customers. The following is my submission for the assignment of the Simulation (MATH11028¹) course.

The deliverable of this course was to simulate a simple bank queue with three scenarios. The first scenario was single server (employee serving customers) and an exponential random time of arrival and an Erland random number of completion. Both these numbers were derived from Java’s Random class. The myRandom class, included in the public Git repository², and it was created by my lecturer Andreas Grothey. It is basically an implementation of generating an erlang variate Erl(λ, μ) and en exponential one Exp(μ), from a uniformly distributed Gaussian distribution³.

For this assignment, different scenarios for managing a bank queue were requested from the bank manager. The first scenario involved a queue with single server with an exponential distribution for arrival. The average of customers arriving was 19 per hour. The completion times were given by an Erlang distribution. The number of customers served per hour were 20. The day in all simulations was taken to last 8 work-hours. The result of this scenario was that out of 164 arrivals, all customers were served with 85.4% (140 customers) having waited more than 6 minutes in the queue.

The second scenario  involved extending the queue to multiple servers.  An ArrayList was used to keep track of servers’ status and time of completion.  This scenario produced non optimal results, since there ware 22.8% of customers waiting more than 6 minutes. The maximum queue for this scenario was 17 customers.

The third and final scenario was to implement different average numbers of customers for each of the 8 hours in the work day. The additional decision was to implement a strategy of opening and closing server positions when demand increases.  The demand criterion was selected to be the length of the queue at any given time. There were different scenarios to try out, but the sole setting that met the constraint was when the queue did not exceed 6 customers.

The plots above were generated with R⁴ after out-putting the Java stdout to a file called “fraction”. From there the contents of that file were passed though awk in order to sieve the fraction values from all other data into a file called “fraction2″.  The latter file was used as the dataset in R. The history for the commands used in R are here.

The figure above shows at the top part, the procession of the fraction throughout the day. The blue horizontal line is the constraint ceiling of 5% (0.05).

The results for this scenario were run over 100 trial iterations, in order to calculate an average value for the percentage of customers that wait more than 6 minutes in the queue.  This value was resulting at 3.2% ;well within the limit. If the queue tolerance was extended to being more than 7, the average fraction was outside the limit of 5%.

The two diagrams show the time-series of the fraction of customers waiting more than 6 minutes (top) and the number of servers that are engaged at each discrete interval. The red vertical lines are indicating the events at which the hour changes. Therefore they denote the 8 1-hour segment during the day and the servers corresponding to each hour. It is evident that the third, fourth, seventh, and eighth hour of the day demanded the most servers.

An average value of servers per event is also outputted as a means of showing the hours in most need of additional servers. Again the hour slots mentioned above stand out as the highest values.

Avg servers for hour 1 : 2.0

Avg servers for hour 2 : 1.98

Avg servers for hour 3 : 2.641509433962264

Avg servers for hour 4 : 2.323529411764706

Avg servers for hour 5 : 2.25

Avg servers for hour 6 : 2.3307086614173227

Avg servers for hour 7 : 2.580952380952381

Avg servers for hour 8 : 3.278787878787879

The result of the simulation indicates that the bank will require a minimum of two people per hour for all hours of the work day, aside from the end-of-day hour (16.00-17.00) were an additional employee will be required a the service tills.

Footnotes

1.

The structure of Simulation course I sat through. This is here mainly to give people an idea as to what is expected from this course.

Aims:

Many Management problems are far too complex to be solved solely by Mathematical Analysis. Simulation (Monte-Carlo) methods were developed during the Second World War to solve difficult problems in Mathematical Physics. This module shows how discrete event simulation is increasingly used to investigate the behaviour of complex stochastic systems in Management. The course includes hands-on construction of simulation models using the package SIMUL8.

Syllabus summary:

Generating Uniform random numbers using Linear Congruential methods. Tests of randomness including Frequency and Serial tests and Lattice behaviour. Generating Random Variates from distributions: Inversion of the cumulative distribution function and Envelope Rejection methods. Generating variates from standard distributions including normal, gamma, beta, Poisson, binomial. Theory and simulation of Poisson and other birth-death processes. Discrete event simulation: Entity, Event, Activity, Attribute, and Set. Event and Three-Phase approaches to model construction. An example in JAVA. Design and analysis of experiments: Use of random number seeds. Variance Reduction including Antithetic and Control Variates. Analysis of autocorrelated output data. Use of the SIMUL8 package to build and experiment on models of manufacturing and management systems. Project work for continuous assessment part of course.
Back to post.

2.

You can clone, fork, review this code at:

https://github.com/cmdel/Queue-Simulation

git@github.com:cmdel/Queue-Simulation.git (For cloning in Eclipse or the command line)

Back to post.

3.

Back to post

4.

http://www.r-project.org/

Back to post

The drawbacks of this was that I was still spending too much time re-installing all my tools and applications from scratch. Their configurations were safe in the hidden folders in my /home directory. The process of finding everything and keeping track of what I used and manually install them was getting laborious.

So in the true linux spirit, I did what I guess anyone of you reading this post would do. I just created a small script to get me back to how my system used to be. This script will make a backup copy of the existing repository list first and then replace it will a list that will include all the necessary entries to install the software I need [File1:Lines2-3].

Then a few necessary encryption keys are installed so that certain repositories can be authenticated [File1:Lines4-8]. And from then on there’s a very long line with every single piece of binary I want installed [File1:Line10]. If I find I’m using a new application very often I will add it in the list here and add its repositories in the scripts repository list.

A few statistics packages I use are then added, as well as the libraries to decrypt and play DVDs [File1:Lines13-15].

A few screenlets I’m using are next [File1:Lines20-22].

It also gets my .rc files from github [File1:Lines26-27].

Finally it will install wine for the windows application I want to run, e.g. in this case Spotify [File1:Lines31].

And this is pretty much it. The script is kept simple so that it’s easy to extend to your needs. Just add the application’s repository in the list, add any optional keys to your keychain, and finally add the installation name of the application to the list to be installed. Conversely, just remove anything you don’t want to install.

This script is put here for anyone wanting something similar and not knowing where to start. Copy and share, no attribution needed. Also implied in this that this script comes as it is. I’m not making claims that it won’t hurt your installation. I’ve tested it on  many occasions and it works. The system it’s supposed to be used anyway is a manilla one so I guess it won’t matter if you mess it up. You can just re-install it.

File1:

#!/bin/bash
cp /etc/apt/sources.list /etc/apt/sources.list.bak
cp maverick.sources.list /etc/apt/sources.list
add-apt-repository ppa:tiheum/equinox
add-apt-repository ppa:am-monkeyd/nautilus-elementary-ppa
add-apt-repository ppa:bisigi/dev # Needs to change to /ppa when it's available
apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 2EBC26B60C5A2783 F9A2F76A9D1A0061 A040830F7FAC5991 16126D3A3E5C1192 DCF9F87B6DFBCBAE D6FDC9D0 FC5D50C5 2D9A3C5B 0AAFAD78 A69241F1 D67FC6EAE2A11821
apt-key adv --recv-keys --keyserver pgp.mit.edu 5044912E #dropbox key
apt-get --yes --quiet update && apt-get --yes --quiet dist-upgrade
apt-get --yes --quiet install clamav clamtk build-essential checkinstall cdbs devscripts dh-make fakeroot libxml-parser-perl check avahi-daemon sun-java6-jre sun-java6-plugin equivs vlc mplayer libxine1-ffmpeg gxine mencoder mpeg2dec vorbis-tools id3v2 mpg321 mpg123 libflac++6 ffmpeg libmp4v2-0 totem-mozilla icedax tagtool easytag id3tool lame nautilus-script-audio-convert libmad0 libjpeg-progs libquicktime1 flac faac faad sox ffmpeg2theora libmpeg2-4 uudeview flac libmpeg3-1 mpeg3-utils mpegdemux liba52-0.7.4-dev  ubuntu-restricted-extras gsfonts gsfonts-x11 flashplugin-nonfree simple-ccsm wine playonlinux glipper unace rar unrar zip unzip p7zip-full p7zip-rar sharutils uudeview mpack lha arj cabextract file-roller chromium-browser chromium-browser-l10n acroread dia quanta gnome-do docky yakuake guake backintime-common backintime-gnome trash-cli git-core bisigi-themes community-themes nautilus-dropbox elementary-theme elementary-icon-theme elementary-wallpapers conky tint2 screenlets python-simplejson python-twitter compiz-fusion-plugins-extra screen zsh vim htop mutt irssi deja-dup faenza-icon-theme appmenu-gtk indicator-appmenu indicator-applet-appmenu gstreamer0.10-ffmpeg gstreamer0.10-fluendo-mp3 gstreamer0.10-gnonlin gstreamer0.10-pitfdll gstreamer0.10-sdl gstreamer0.10-plugins-bad-multiverse gstreamer0.10-schroedinger gstreamer0.10-plugins-ugly-multiverse totem-gstreamergstreamer0.10-ffmpeg gstreamer0.10-fluendo-mp3 gstreamer0.10-gnonlin gstreamer0.10-pitfdll gstreamer0.10-sdl gstreamer0.10-plugins-bad-multiverse gstreamer0.10-plugins-ugly-multiverse totem-gstreamer gstreamer-dbus-media-service gstreamer-tools

# Statistical tools
apt-get --yes --quiet install r-base-dev octave
wget http://downloads.sourceforge.net/project/sofastatistics/sofastatistics/0.9.24/sofa_0.9.24-1_all.deb?r=http%3A%2F%2Fwww.sofastatistics.com%2Fdownloads.php&ts=1295372877&use_mirror=switch
dpkg -i sofa_0.9.24-1_all.deb
# DVD libraries
apt-get --yes --quiet install libdvdread4 && /usr/share/doc/libdvdread4/./install-css.sh
# Install maintenance
apt-get clean
apt-get autoclean
apt-get autoremove
# Screenlets
wget http://twitterscreenlet.googlecode.com/files/TwitterScreenlet-0.3.1.tar.gz && tar vxzf TwitterScreenlet-0.3.1.tar.gz && mkdir -p ~/.screenlets && mv Twitter ~/.screenlets && rm -rf TwitterScreenlet-0.3.1.tar.gz
wget http://gnome-look.org/CONTENT/content-files/102890-FolderView.tar.gz && tar vxzf 102890-FolderView.tar.gz  --directory ~/.screenlets && rm -rf 102890-FolderView.tar.gz
wget http://gnome-look.org/CONTENT/content-files/71034-3-pingus.tar.gz && tar vxzf 71034-3-pingus.tar.gz  --directory ~/.screenlets && rm -rf 71034-3-pingus.tar.gz*
#bash /usr/share/elementary-desktop/configure.sh # Elementary configuration

# .rc files from repo
wget -q --no-check-certificate http://github.com/cmdel/rc-configs/raw/master/.screenrc
wget--no-check-certificate -q http://github.com/cmdel/rc-configs/raw/master/.zshrc
# Create the file to keep all known hosts for autocomplete
touch ~/.hosts
# Wine configuration
su - cmd -c 'winecfg && wget http://www.spotify.com/download/Spotify%20Installer.exe && wine Spotify\ Installer.exe' && chsh `whoami` -s /bin/zsh && exit

This is the file with the repositories that are necessary.

File2:

# Ubuntu Maverick Meerkat 10.10 Sources list
#
# Repository List based on standard maverick with many extra packages
#
# If you get errors about missing keys, lookup the key in this file
# and run these commands (replace KEY with the key number):
#
# sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com KEY

# Ubuntu supported packages
deb http://fi.archive.ubuntu.com/ubuntu/ maverick main restricted multiverse universe
deb http://fi.archive.ubuntu.com/ubuntu/ maverick-backports main restricted universe multiverse
deb http://fi.archive.ubuntu.com/ubuntu/ maverick-updates main restricted multiverse universe
deb http://security.ubuntu.com/ubuntu maverick-security main restricted universe multiverse
deb http://security.ubuntu.com/ubuntu maverick-proposed main restricted universe multiverse
deb-src http://fi.archive.ubuntu.com/ubuntu/ maverick main restricted multiverse universe
deb-src http://fi.archive.ubuntu.com/ubuntu/ maverick-backports main restricted universe multiverse
deb-src http://fi.archive.ubuntu.com/ubuntu/ maverick-updates main restricted multiverse universe
deb-src http://security.ubuntu.com/ubuntu maverick-security main restricted universe multiverse
deb-src http://security.ubuntu.com/ubuntu maverick-proposed main restricted universe multiverse

#Canonical 'partner' Repository
deb http://archive.canonical.com/ubuntu maverick partner
deb http://archive.canonical.com/ubuntu maverick-backports partner
deb http://archive.canonical.com/ubuntu maverick-updates partner
deb http://archive.canonical.com/ubuntu maverick-security partner
deb http://archive.canonical.com/ubuntu maverick-proposed partner
deb-src http://archive.canonical.com/ubuntu maverick partner
deb-src http://archive.canonical.com/ubuntu maverick-backports partner
deb-src http://archive.canonical.com/ubuntu maverick-updates partner
deb-src http://archive.canonical.com/ubuntu maverick-security partner
deb-src http://archive.canonical.com/ubuntu maverick-proposed partner

#Third Parties Latest Software
deb http://extras.ubuntu.com/ubuntu maverick main
deb-src http://extras.ubuntu.com/ubuntu maverick main

#medibuntu ********* Must chage to Maverick *******
# sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 2EBC26B60C5A2783
deb http://packages.medibuntu.org/ maverick free non-free
deb-src http://packages.medibuntu.org/ maverick free non-free

#opera
# sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com F9A2F76A9D1A0061
deb http://deb.opera.com/opera/ lenny non-free

#google
# sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com A040830F7FAC5991
deb http://dl.google.com/linux/deb/ stable non-free main

#Skype
deb http://download.skype.com/linux/repos/debian/ stable non-free

#Back in Time -
deb http://ppa.launchpad.net/bit-team/stable/ubuntu maverick main
deb-src http://ppa.launchpad.net/bit-team/stable/ubuntu maverick main

#Dropbox
deb http://linux.dropbox.com/ubuntu maverick main
deb-src http://linux.dropbox.com/ubuntu maverick main

# Elementary Desktop
deb http://ppa.launchpad.net/elementaryart/ppa/ubuntu maverick main restricted multiverse universe
deb-src http://ppa.launchpad.net/elementaryart/ppa/ubuntu maverick main restricted multiverse universe

deb http://ppa.launchpad.net/elementaryart/elementarydesktop/ubuntu maverick main restricted multiverse universe
deb-src http://ppa.launchpad.net/elementaryart/elementarydesktop/ubuntu maverick main restricted multiverse universe

# Statistical Analysis Packages
deb http://cran.stat.ucla.edu/bin/linux/ubuntu maverick main

Attributions

Image by XKCD

After installing a few applications to the iMac, which is an understatement in itself!, I realised that the hard disk was becoming increasingly audible at certain intervals. At the time the machine felt sluggish and it really disrupted the quiet environment I’m used to as of yet. The OS’s Activity Monitor sure enough showed the the high disk activity, but there was no obvious way to determine which was the offending process. After some digging around and some different combinations and a lot of piping I’ve narrowed down to the following command:

alias debug="sudo fs_usage -w -f filesys > ./fs_usage.log; egrep -e read fs_usage.log | less ; egrep -e write fs_usage.log |less; rm -rf fs_usage.log"

If you add this to your shell config file – in my case ~/.zshrc – then the debug command after requesting for the root password will collect some usage data and show sequentially the Reads and Writes that were requested until you broke the debugging( i.e. press C-c).

The last column to the right will contain the name of the process that requested the action. Once you go over the output – keep pressing Space to go to the next page – the most commonly appearing process will most likely be the one increasing the I/O of the HDD. In my case it was the anti-virus being triggered every time I was performing a Spotlight search. Quitting Kasperky’s antivirus from he menu bar seems to have brought back the normal use and quietness.

If the offending process is not easily identified with a web search then close down all critical applications with data you don’t want to lose – you do have a backup policy right? – and start terminating one process at a time:

sudo kill -9 <PID>;

Right now the Sophos antivirus seems to be much better. Any one have prior experience regarding antivirus software for macs?

I’ve recently switched from a Linux laptop to an iMac. The reason for this decision was very simple. I was tired of spending most of my time, fixing and endlessly customizing Linux. Don’t get me wrong though! I loved my time with Linux. I’ve learned quite a lot of stuff “playing” around with it. After a point however, it just isn’t worth my time to spend hours into making Linux work/perform the way I want it to. And then do it all over again when you try to upgrade to a new distro. I ended up doing a fresh install every time and having a script to re-install all the programs I needed. That with a separate /home/ partition provided an easier and safer, believe it or not, solution to an upgrade of Ubuntu. After a point I realized I was spending more time fixing or changing Linux to facilitate me in what I wanted to do, rather than doing the things I needed to be doing.

I needed a machine that literally got out of my way and let me work efficiently and easily. The most appropriate solution was the new iMac. The actual tactile interface is like something I’ve never worked with before. That’s a 27″ monitor that doesn’t make a ‘peep’  when it’s working. And I mean both fan noise and the HDD harassment Windows is performing when doing memory paging. That is a sound I only hear now when I fire up the Parallels version of Windows 7.

I still use the Office 2010 platform. I think it’s one of the most professional tools out there for office work. It’s not strange that Apple has “allowed” this platform to be run natively on Macs. I also use some simulation software that are only available on the windows platform. The Parallels VMs are great at providing a seamless integration of Windows into the OSX and also to have an option to work with Ubuntu when I need it. MacPorts is surprisingly useful for installing CRAN packages in R. On another note, I had to see OSX put itself to sleep to fully appreciate how Sleep should work and how flawed the attempts of Windows and Linux are when trying to do the same thing.

I’ll be returning to this subject in regards with certain productivity apps I’ve found useful up to now. Feel free to share yours as well.