rjmarvin writes "The Apache Software Foundation announced that Spark, the open-source cluster-computing framework for Big Data analysis has graduated from the Apache Incubator to a top-level project. A project management committee will guide the project's day-to-day operations, and Databricks cofounder Matei Zaharia will be appointed VP of Apache Spark. Spark runs programs 100x faster than Apache Hadoop MapReduce in memory, and it provides APIs that enable developers to rapidly develop applications in Java, Python or Scala, according to the ASF."
Please create an account to participate in the Slashdot moderation system
fiannaFailMan writes "Wired reports one mathematician's mission to find love online by data mining from OK Cupid and applying mathematical modeling to optimize his profile(s). His methods included using 'Python scripts to riffle through hundreds of OkCupid survey questions. He then sorted female daters into seven clusters, like "Diverse" and "Mindful," each with distinct characteristics.' But the real work began when he started going on dates."
Recently presented at Linuxconf.au was Glyphy, a text renderer implemented using OpenGL ES2 shaders. Current OpenGL applications rasterize text on the CPU using Freetype or a similar library, uploading glyphs to the GPU as textures. This inherently limits quality and flexibility (e.g. rotation, perspective transforms, etc. cause the font hinting to become incorrect and you cannot perform subpixel antialiasing). Glyphy, on the other hand, uploads typeface vectors to the GPU and renders text in real time, performing perspective correct antialiasing. The presentation can be watched or downloaded on Vimeo. The slide sources are in Python, and I generated a PDF of the slides (warning: 15M due to embedded images). Source code is at Google Code (including a demo application), under the Apache License.
mikejuk writes "A recent xkcd strip has started some deep academic thinking. When AI expert Peter Norvig gets involved you know the algorithms are going to fly. Code Golf is a reasonably well known sport of trying to write an algorithm in the shortest possible code. Regex Golf is similar, but in general the aim is to create a regular expression that accepts the strings in one list and rejects the strings in a second list. This started Peter Norvig, the well-known computer scientist and director of research at Google, thinking about the problem. Is it possible to write a program that would create a regular expression to solve the xkcd problem? The result is an NP hard problem that needs AI-like techniques to get an approximate answer. To find out more, read the complete description, including Python code, on Peter Norvig's blog. It ends with this challenge: 'I hope you found this interesting, and perhaps you can find ways to improve my algorithm, or more interesting lists to apply it to. I found it was fun to play with, and I hope this page gives you an idea of how to address problems like this.'"
On Planet Python, Gregory Szorc asks why many projects continue to support Python releases prior to 2.7, when they are all EOLed (2.4 in 2008, 2.5 in 2011, and 2.6 last October), and Python 2.7 provides a clear upgrade path to Python 3. Quoting: "I think maintainers of Python projects should seriously consider dropping support for Python 2.6 and below. Are there really that many people on systems that don't have Python 2.7 easily available? Why are we Python developers inflicting so much pain on ourselves to support antiquated Python releases? As a data point, I successfully transitioned Firefox's build system from requiring Python 2.5+ to 2.7.3+ and it was relatively pain free." Shortly after posting, other developers responded with their reasons for using older Python releases. First, Rob Galanakis of CCP (EVE Online) explains the difficulties involved in upgrading a mature commercial project embedding Python. Nathan Froyd adds "I think this list of reasons to upgrade misses the larger point in providing software for other people: You do not get to tell your users what to do. ... Maybe those users don’t have sufficient control over their working environments to install a new version of Python. ... Maybe those users rely on certain APIs only available in older versions of Python and don’t wish to take an indeterminate amount of time to rewrite (retest, recertify, etc. etc.) their software. ... Maybe those users are just rationally lazy and don’t want to deal with downloading, configuring, and installing a new version of Python, plus dealing with inevitable fallout, when the old version has worked Just Fine for everything else."
New submitter KDE Community writes that the KDE project has released KDevelop 4.6.0 as the latest version of the free and open source integrated development environment. "KDevelop 4.6.0 improves debugging support with GDB. The GDB integration improvements include some operations now going into effect immediately rather than needing to re-run the program, improved debugging from external terminals, and a CPU registers toolview. KDevelopers' CPU registers toolview also allows for showing and editing all user-mode registers and general purpose flags for x86/x86_64/ARMv7 platforms. Other KDevelop 4.6.0 changes include greater language support within the PHP plug-in, Python language support improvements, more C++11 language support, improved project management, and a clean-up to the IDE's user-interface."
shutdown -p now writes "Coming from the team that had previously brought you Python Tools for Visual Studio, Microsoft has announced Node.js Tools for Visual Studio, with the release of the first public alpha. NTVS is the official extension for Visual Studio that adds support for Node.js, including editing with Intellisense, debugging, profiling, and the ability to deploy Node.js websites to Windows Azure. An overview video showcases the features, and Scott Hanselman has a detailed walkthrough. The project is open source under Apache License 2.0. While the extension is published by Microsoft, it is a collaborative effort involving Microsoft, Red Gate (which previously had a private beta version of similar product called Visual Node), and individual contributors from the Node.js community."
cold fjord writes with this report from The Telegraph: "The original members of Monty Python will reunite more than 30 years after the comedy troupe last worked together. John Cleese, Terry Gilliam, Terry Jones, Eric Idle and Michael Palin will officially announce their reformation at a London press conference on Thursday. The five surviving members have reportedly been in months of secret talks about getting the Flying Circus back on the road. The reunion comes after several failed attempts to reform by the group. However, according to The Sun, the surviving members realised 'it was now or never,' and had decided to embark upon 'a fully-fledged reunion.'" Related stories include this commentary, one take on the best of Python and this negative reaction, too.
An anonymous reader points out The Register's report that Wi-Fi security expert Cédric 'Sid' Blancher has died as the result of a skydiving accident. "Among other things, the 37-year-old Blancher was a sought-after speaker on WiFi security, and in 2005 published a Python-based WiFi traffic injection tool called Wifitap. In 2006, while working for the EADS Corporate Research centre, he also put together a paper on how to exploit Skype to act as a botnet." Some of Blancher's skydiving videos are posted to Vimeo; clearly, it's something he was passionate about.
jfruh writes "Most day-to-day programmers have only a general idea of how compilers transform human-readable code into the machine language that actually powers computers. In an attempt to streamline applications, many compilers actually remove code that it perceives to be undefined or unstable — and, as a research group at MIT has found, in doing so can make applications less secure. The good news is the researchers have developed a model and a static checker for identifying unstable code. Their checker is called STACK, and it currently works for checking C/C++ code. The idea is that it will warn programmers about unstable code in their applications, so they can fix it, rather than have the compiler simply leave it out. They also hope it will encourage compiler writers to rethink how they can optimize code in more secure ways. STACK was run against a number of systems written in C/C++ and it found 160 new bugs in the systems tested, including the Linux kernel (32 bugs found), Mozilla (3), Postgres (9) and Python (5). They also found that, of the 8,575 packages in the Debian Wheezy archive that contained C/C++ code, STACK detected at least one instance of unstable code in 3,471 of them, which, as the researchers write (PDF), 'suggests that unstable code is a widespread problem.'"
An anonymous reader writes "Investment firm Knight Capital made headlines in 2012 for losing over $400 million on the New York Stock Exchange because of problems with their algorithmic trading software. Now, the owner of a Python programming blog noticed the release of a detailed SEC report into exactly what went wrong (PDF). It shows how a botched update rollout combined with useless or nonexistent process guidelines cost the company over $172,000 a second for over 45 minutes. From the report: 'When Knight used the Power Peg code previously, as child orders were executed, a cumulative quantity function counted the number of shares of the parent order that had been executed. This feature instructed the code to stop routing child orders after the parent order had been filled completely. In 2003, Knight ceased using the Power Peg functionality. In 2005, Knight moved the tracking of cumulative shares function in the Power Peg code to an earlier point in the SMARS code sequence. Knight did not retest the Power Peg code after moving the cumulative quantity function to determine whether Power Peg would still function correctly if called. ... During the deployment of the new code, however, one of Knight's technicians did not copy the new code to one of the eight SMARS computer servers. Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added. Knight had no written procedures that required such a review.'"
angry tapir writes "Mining small details from Facebook has become even easier with Graph Search, the site's new search engine that returns personalized results from natural-language queries. Graph Search granularly mines Facebook's vast user data: where people have visited, what they like and if they share those same preferences with their friends. 'FBStalker' is a Python script debuted at the Hack in the Box security conference in Kuala Lumpur. In its current form, FBStalker runs in the Chrome browser on OS X, entering queries into Facebook's Graph Search and pulling data. Even if a person's profile is locked down to strangers, their friends' open profiles can be examined, giving an indication, for example, who the person may be close with. FBStalker uses Graph Search to find photos in which two people are tagged in, comments on profiles and more."
ananyo writes "An offshoot of Mozilla is aiming to discover whether a review process could improve the quality of researcher-built software that is used in myriad fields today, ranging from ecology and biology to social science. In an experiment being run by the Mozilla Science Lab, software engineers have reviewed selected pieces of code from published papers in computational biology. The reviewers looked at snippets of code up to 200 lines long that were included in the papers and written in widely used programming languages, such as R, Python and Perl. The Mozilla engineers have discussed their findings with the papers’ authors, who can now choose what, if anything, to do with the markups — including whether to permit disclosure of the results. But some researchers say that having software reviewers looking over their shoulder might backfire. 'One worry I have is that, with reviews like this, scientists will be even more discouraged from publishing their code,' says biostatistician Roger Peng at the Johns Hopkins Bloomberg School of Public Health in Baltimore, Maryland. 'We need to get more code out there, not improve how it looks.'"
theodp writes "Nate West has a nice essay on the importance of whimsy in learning to program. "It wasn't until I was writing Ruby that I found learning to program to be fun," recalls West. "What's funny is it really doesn't take much effort to be more enjoyable than the C++ examples from earlier...just getting to write gets.chomp and puts over cout > made all the difference. Ruby examples kept me engaged just long enough that I could find Why's Poignant Guide to Ruby." So, does the future of introductory computer programming books and MOOCs lie in professional, business-like presentations, or does a less-polished production with some genuine goofy enthusiasm help the programming medicine go down?"
angry tapir writes "Much of the urban vistas of Man of Steel, Cars 2 and the horrible remake of Total Recall were not modelled by hand. Instead they relied on a product called CityEngine, which is more typically associated with local government bodies' urban planning and urban design. The software procedurally generates cities using scripts written in a Python-like language. The next version of CityEngine, coming out next month, will incorporate an SDK so third-party developers can use parameter-defined procedural generation of urban environments in their own applications. CityEngine's product manager talks about the upcoming version, how it's being used at the moment, and plans to incorporate augmented reality in it."
cold fjord sends news that a study by Coverity has found open-source Python code to contain a lower defect density than any other language. "The 2012 Scan Report found an average defect density of .69 for open source software projects that leverage the Coverity Scan service, as compared to the accepted industry standard defect density for good quality software of 1.0. Python's defect density of .005 significantly surpasses this standard, and introduces a new level of quality for open source software. To date, the Coverity Scan service has analyzed nearly 400,000 lines of Python code and identified 996 new defects — 860 of which have been fixed by the Python community."
rjmarvin writes "Two developers were able to successfully reverse-engineer Dropbox to intercept SSL traffic, bypass two-factor authentication and create open-source clients. They presented their paper, 'Looking inside the (Drop) box' (PDF) at USENIX 2013, explaining step-by-step how they were able to succeed where others failed in reverse-engineering a heavily obfuscated application written in Python. They also claimed the generic techniques they used could be applied to reverse-engineer other Frozen python applications: OpenStack, NASA, and a host of Google apps, just to name a few..."
Last week you had a chance to ask Guido van Rossum, Python's BDFL (Benevolent Dictator For Life), about all things Python and his move to Dropbox. Guido wasted no time answering your questions and you'll find his responses below.
Guido van Rossum is best known as the creator of Python, and he remains the BDFL (Benevolent Dictator For Life) in the community. The recipient of many awards for his work, and author of numerous books, he left Google in December and started working for Dropbox early this year. A lot has happened in the 12 years since we talked to Guido and he's agreed to answer your questions. As usual, ask as many as you'd like, but please, one question per post.
mikejuk writes "Is it possible that we have been wasting our time typing programs. Could voice recognition, with a little help from an invented spoken language, be the solution we didn't know we needed? About two years ago Tavis Rudd, developed a bad case of RSI caused by typing lots of code using Emacs. It was so severe that he couldn't code. As he puts it: 'Desperate, I tried voice recognition'. The Dragon Naturally Speaking system used by Rudd supported standard language quite well, but it wasn't adapted to program editing commands. The solution was to use a Python speech extension, DragonFly, to program custom commands. OK, so far so good, but ... the commands weren't quite what you might have expected. Instead of English words for commands he used short vocalizations — you have to hear it to believe it. Now programming sounds like a conversation with R2D2. The advantage is that it is faster and the recognition is easier — it also sounds very cool and very techie. it is claimed that the system is faster than typing. So much so that it is still in use after the RSI cleared up."