Ask HN: Any active COBOL devs here? What are you working on?

Very old coder here. Wrote COBOL to help Atari add features to a inventory processing system to account for the fact that "inventory" intially was items received at the loading dock, fork-lifted to the shipping dock and shipped. So "inventory" needed to be booked immediately as sales. Now I dabble mostly with Python and JS/HTML. My memory of the Atari gig was that the most critical part was the CICS code. There was just one guy who knew enough to setup the CICS. If he got hit buy a bus... Well after about a year, the bus would not have mattered. Atari buried millions on unsold carts, and I went from working in a beautiful office complex next to Great America Park, to a windowless basement closet somewhere near Mountain View now making changess because the "forklift inventory" version was no longer needed. I know this is a bit off topic, but "COBOL" was the into I needed.

My mother and her husband are COBOL devs for a US state government. She works on the health insurance side for teachers and other state employees. Think claim processing.

Lots of batch jobs running at night. Their alert system is an actual human who calls my mom when jobs fail in the middle of the night.

It's high paying for the city they live in, but not high paying for software development. They will both have full retirement and healthcare for life, assuming the government can fulfill it. They are both fully remote since COVID too.

She's also worked for state lottery, teacher's retirement system and DOT.

edit: she says they have a SQL database, but mostly store in IBM IMS

Is it entirely maintenance or does she also build new stuff with COBOL?

By state, you mean US state, right?

Correct. Updated.

Very likely.

I work with a lot of COBOL dinosaurs in the bank, I often like to watch them work on their 16-colors IBM z/OS host terminals, it's quite mesmerizing. Sometimes they show me some interesting code that was written before I was alive (I'm 36), or tell me stories about big mainframe incidents in the '80s, where they would get called in the middle of the night and flown to a different country to fix a bug because there was no remote desktop back then.

Damn mainframe people flaunting their 16 colors like they're a peacock or something. Shit. We only had one color (and one absence of a color) and that was good enough.

> I often like to watch them work on their 16-colors IBM z/OS host terminals, it's quite mesmerizing

They really are. I had a parttime coworker who moonlighted some mainframe job and he often had another laptop on his desk connected to a z/OS terminal. He would show me some of the jobs and code occasionally too, really fascinating stuff, and he was quite good at it and could navigate quickly.

PuTTY into Linux and you're in 16 colors.

My Linux terminal is 256 colors. Two hundred fifty six! That's like, every color!

<16-colors IBM z/OS host terminal

This hasn't been virtualized?

I’m not a COBOL dev but I work with mainframes(z/OS). Most COBOL applications I’ve seen have been banking and insurance related with few exceptions. Most of them either run as a series of batch jobs or via transaction managers like IMS and CICS. Backends are usually sequential files(we call them datasets),DB2,VSAM(Virtual Storage Access Method) or DL/1(hierarchical DB that’s part of IMS). Quite a few places I’ve seen have run IBM MQ as well.

If changes are made to these systems it’s often due to changes in regulation or driven by changes in the business(new financial products being offered etc.

Off-topic: I’ve seen quite a few mainframe related posts on HN fly by over the years. I’ve been meaning to create an account and participate but I’ve only gotten around to it just now.

Thanks for your insight - it’s comments like this which make HN a place worth visiting every day.

And welcome!

Dumb question: mainframes and z/OS look interesting, how does one get started with learning about those systems and those environments ?

There's an emulator called Hercules[1] that lets you run (some) mainframe stuff on a PC. There are limits to what you can run on it though, mostly due to licensing issues with IBM.

You can also look at the IBM Redbooks site[2]. Search for terms like Z/OS, MVS, CICS, DB/2, etc. and you'll find a lot of IBM books, whitepapers (well, they call them redpapers, but whatever) and so on.

[1]: https://en.wikipedia.org/wiki/Hercules_(emulator)

[2]: https://www.redbooks.ibm.com/

I'm not a mainframe programmer, but coming from x86 land I was very curious. I really learned a lot from the IBM Coursera "Intro to Mainframe" since none of my experience really applied it was tough. It had a real shell account to practice with though.

https://www.coursera.org/professional-certificates/ibm-z-mai...

Also the MOSHIX mainframe YouTube channel has a lot of info, and helped me setup HERCULES emulator for my own little mainframe experience.

http://www.hercules-390.org/

Not a dumb question at all! In Europe I’ve seen a few training programs held by companies looking to get new talent in to learn from the older techs. Browse around and see if any companies around you have something like that.

There are some free resources available that will allow you to get training but I haven’t tried them myself. IBM Z Xplore is worth a look as an example: https://www.ibm.com/products/z/resources/zxplore

I hope you find a way in, more mainframe developers and sysadmins(often called systemsprogrammers in the mainframe niche) are always needed.

Edit*: Spelling and grammar

nice, welcome to the party

Thank you!

Not a COBOL developer, but working at a sizeable bank I witnessed the phasing out of their mainframes and AS400 systems. They ran some critical systems, both in retail and wholesale banking. They either converted to java, and optimized that code, but some COBOL code from the mainframe, and all of the AS400 stuff was converted into Micro Focus COBOL, which runs on Windows, which could be hosted on our Private Cloud. I worked on helping them migrate to our cloud infra, which was an interesting exercise. There was a very tangible cultural gap between the people maintaining and developing these applications and the rest of the organization.

Can you describe the cultural gap? I haven't really met these folks in the wild, so I'm curious what the programmers of yore were like.

In my experience, it's usually lack of awareness about modern security risks, and lack of familiarity with modern infrastructure paradigms. The latter really isn't a problem since these systems are usually standalone, but the former does become a problem - they often are from a time where this just wasn't something to consider. As a result, these legacy systems are often using default passwords, have tons of crazy stuff exposed to the network, and are comprised of custom code written specifically for the business purpose (so the documentation is only as good as what they made).

On the other hand, these guys generally write pretty neat, lean code that is quick, reliable, and directly responsive to the business. The really fun thing is watching the users fly through the keyboard-only screens, sometimes with muscle memory that is faster than the terminal emulator can update - they're literally working ahead of the screens.

In my experience mainframes at financial institutions are hidden behind IBM middleboxes that are specifically designed to obviate the infrastructure risks. It's a classic example of a company selling you both the problem and solution.

Oh yes, I remember that when we swapped out a bunch of terminals at an airline.. The users complained it was all way too slow on the new Windows machines with MS SNA server in between... I was wondering what it was all about, as a young and very naive dropout from uni on his first IT job. When I came down, this dude was banging on his keyboard and after some time stopped, pointed at the screen and you could see it slowly catching up, screen by screen.. He showed me the directly connected version next. I learned something that day.

I would say these people were in a relationship with the mainframe, if that makes sense. And also having worked at IBM in the past where I sat adjacent to the mainframe support team for Business Services, I totally get it. Mainframes are awesome if you ask me, and in a sense we have been trying to reinvent a lot of its goodness with "commodity" x86 hardware.

From a technical-cultural perspective it was mostly sulkiness, and a complete and utter lack of embracing the paradigms of distributed computing. Also, like most internal clouds, there were plenty of issues as it was. Initially they just tried to replace mainframe application components 1:1 onto VMs in whatever way and whenever anything was <100% reliable they complained that our cloud was not able to do it. I had to explain in a very harsh way, under a lot of pressure (I believe not hitting the deadline of switching off the mainframes meant renewal for a year at 40 Mil.. or thereabouts) the realities of "cloud".

The developers I spoke with in that time though, were very much the opposite of the move fast breaking things crowd. Intelligent, but also narrow minded I would say.

Not Recently. 2010ish was working for an insurance processor that was actively writing thousands of lines and executing on z/OS with no plans to migrate.

I was part of team that was writing web applications that needed to call z/OS transactions. The IBM solution was to use their Transaction Gateway product, which cost a ton, and was slow as shit. We developed a framework that used annotations on the Java Side to map COBOL Records to Java Objects and invoke transactions over a TCP socket. Learning how to pack decimals and convert encodings was pretty cool. We ended up with a framework that was at least a zillion times faster than the IBM solution. Left that job though as the company was is distress and was losing customers (health plans). They eventually folded.

I haven't worked in COBOL, but I've worked with it.

This was around 1999, and I was building a system for configuring and ordering custom PC's at a large distribution company. One of the feature requirements was that we display inventory over the various options. (ie: There are 373 20G disks in stock, but only 12 30G disks in stock). The idea was that this would let a customer ordering 200 machines know that they should pick the 20G disk if the wanted it now.

The way inventory at this company was done was via two systems. There was a normal SQL database that had access to a daily snapshot of inventory taken from a mainframe that always had the up to date data. With the mainframe taking a while to process queries, we used the less current SQL database for the majority of the UI, but took the time to query the mainframe once a customer was in the shopping cart. Customers might see a change during the flow, but it would at least let them see the most current data prior to committing to a purchase.

The mainframe query itself was implemented by someone else as a COBOL job that produced the required inventory numbers. From my point of view, it was just a limited sort of query conducted over a specialized JDBC driver. (Not necessarily the weirdest aspect of that design.... for a variety of reasons, we did the UI in ASP/VBScript, the back end using Microsoft's JVM invoked via COM Automation, and the SQL database link was via a dubious use of a JDBC/ODBC bridge to connect to SQL Server. It all worked, but not the most solid architecture.)

==

My only other mainframe experience was working as an intern for a utility company a few years prior (1991-1992). They used CDC Cyber mainframes to run the power grid, using something like 4MM lines of FORTRAN code. The dispatchers themselves interfaced to the system using consoles with 4 19" color displays running at 1280x1024. Heady stuff for 1991. (The real time weather radar screen was cool too, in an age before the internet made it common place.)

Met one close to retirement who worked on a ERP system in the food processing industry. Nightly batch jobs would trigger orders from their suppliers, customer service would enter new orders. Two SAP migrations already failed, costing the company millions. All company process knowledge was in code, database fields have been repurposed (but no renamed, too much work), feature development stop long time ago. In parallel a new system was built in-house (no longer trusting external consultants) and his job was explaining what the system does. Probably well paid but he didn't seem to care, he just wanted to work less and retire on good terms.

I grew to like migration projects like that.

Currently working on migration of 30yo ERP without tests in Progress to Kotlin+PostgreSQL.

AI agents don’t care which code to read or convert into tests. They just need an automated feedback loop and some human oversight.

I would argue that they need heavy human oversight

A note-taking app.

I've had a bunch of recent projects reverse-engineering old COBOL code, in financial services.

Mostly to figure out the best way to replace the old systems with something newer, so not really as a "COBOL dev", though.

Anything in particular you're replacing them with generally?

I heard a story about replacing COBOL with JavaScript ... and my skin still crawls thinking about it.

There's surprisingly a lot of finance related jobs in TypeScript. I wonder what libraries they are using for money management.

Indeed, I've worked on billing system that relied heavily on pure JavaScript. Not even modern flavors with map/reduce, etc. - ECMAScript 5. It worked surprisingly well and our bottleneck wasn't the runtime but rather the databases we were constantly upserting to.

It sounds kinda crazy but with good change control, documentation, good relationship with the ETL team - it was pretty maintainable.

Heh I worked one of these. We handled arithmetic in the DB tho. Lot of PL/pgSQL running under the typescript, TS was more of like a middleware or API layer for things that could change more frequently. Finance code is to some extent transcribing of regulation & law into code and we kept all that in postgres.

I met a dev who's mom had been working on legacy banking systems her whole career. She had started in the eighties and she still did some urgent jobs at a crazy rate despite officially having retired.

My stepmom who retired five years ago, did COBOL dev as part of her banking job until 2002ish and then she was full-time management track. In her bank, most of the work had been integrated with Java, and the Java was done by outsourced Indian teams. At the time she retired she felt the Indian teams had been failing for years to meet objectives, and finally management was seeing it. Additionally everybody who knew the COBOL side of things was retiring at the same time as she was and she did not want to know what the system would look like in five years.

My mother used to teach Cobol back in the 80’s in Brazil but later she transitioned into management and haven’t touched a line of code for more than 30 years, she can’t even speak english wtf

Credo!

I have written cobol in the past. I worked at a financial company that had a decent amount of cobol devs around, with the primary database being DB2. The code is mostly financial transactions and record updates, basically CRUD code. Cobol was essentially the backend with the front end being Java and Javascript/Angular.

Government. About 25% of my job. No day to day mainframe development, but we do need to update some logic for new policies and regulations.

There’s not much maintenance work. There are very few bugs, as the core applications have been running for decades, most come up with interactions to external services.

Any major development projects are only in service of lower overall COBOL development time, like transitioning some business logic to database updates.

And there is a decommission plan for the mainframe, so plenty of work helping that team.

What I'm wondering: Are the salaries high? Not just because you've been employed at the job for a long time with regular raises, but because it's hard to find developers.

No the salaries aren't high. They are typically lower than other software engineer salaries. There are a large number of contractors from Indian consulting companies with "experience in cobol" to make run of the mill cobol cheap enough.

The very high salaries you hear about sometimes are always for VERY specific mainframes that are extremely old with a lot of quirks, and are usually being paid to consultants.

A while back I came across job listings for a COBOL consultancy near me that only seems to hire fresh grads for well below market rate (not much higher than retail/restaurant jobs - this is in a cheaper part of the US). They promised to train their employees from the ground up and implied that COBOL knowledge would set them up for a really profitable career. It seems like they were taking advantage of the common advice: "just become a COBOL developer, it pays well because nobody wants to use COBOL!" But I'm skeptical that someone coming out of that consultancy with 2 or 3 years of experience in nothing but COBOL would do well on the job market.

The places I know that use (or used cobol 5 years ago) were all in hiring freezes for cobol developers and were trying to get off of it as much as they could (no new development, only maintenance, etc). I don't think its a surefire bet.

> There are a large number of contractors from Indian consulting companies with "experience in cobol" to make run of the mill cobol cheap enough.

Seeing the horrible performance from Indian offshore firms with modern languages I cannot imagine the mess they make with legacy languages like Cobol. Or is it the other way around?

Corps operate despite inefficiency not because of efficiency. LLC protection and market control are everything.

But the code still has to work. LLC's and other corporate structures only protect the owners if the company goes bankrupt, which it will if its systems stop working.

Ditto with market control, it's not some permanent crown you achieve. Companies have to keep performing to keep their market share.

E.g., if you opened an account at a major bank, and your transactions started failing, would you keep banking there?

> E.g., if you opened an account at a major bank, and your transactions started failing, would you keep banking there?

A lot of people who land in that situation do continue banking there since they are either tied into that bank through loans/debt, or lack the time/energy to move elsewhere.

I suppose it varies by country?

In my country, no, COBOL jobs aren't well paid. They are below average.

This is in my past, so no COBOL recently but in one of my first consulting gigs I wrote several business programs in RPG III. Which is the nightmare that you might imagine it is. Think plugboards. Halfway through the IBM guy came by and installed COBOL and wow what a difference. My last act was to recommend that they get a computer department. And so Tom Monaghan called IBM

My brother works with COBOL for a bank here in Brazil, he is young (in his 20s), started before finishing his degree. Pay is poor, hours are insane, he is overworked as hell, and anything “modern”, like git, is out of the question.

He’s trying to learn Go now and modernize himself to see if he can get out. I’m trying to help as much as I can. Hopefully, he’ll land a job somewhere else this year.

Global Shop ERP is written in Visual Cobol, believe it or not. Supposedly they are actively rewriting the eight million lines of code to C#.

I know some COBOL devs. They work in a bank.

They wouldn't hang around here though.

Where would they hang around? Curious to learn about other programming/tech communities

They aren't really into tech outside of work.

Same, lately.

... or, arguably, inside of work. :)

Not me, but not even 5 years ago one of my friends was working on Cobol codebase running on IBM zOS. It did not last long, but as far as know they had a decent time.

Maybe the COBOL devs aren’t here.

Ex COBOL dev here. Started at NCR on COBOL for minis and mainframes. Then built stuff on Wang VS COBOL for ERP systems until 93. Quite a few other languages since then but been a Product Manager for past 20+ years.

Same here, I worked on Wang VS COBOL too. IMO, it had the best TUI Screen generation Tools I have ever seen.

Towards the end, I worked on a project to port Wang VS OS and its COBOL to AIX. I was tasked finding issues with COBOL Programs we had on the VS. It was a good environment, but Wang went CPT 11 before it was ready :) It was rather close to being complete.

HN is a place where everyone is, so it's reasonable to assume that some Cobol devs may also lurk here.

I am here; so I can attest to HN being a place for everyone.

Perl devs too. There are literally dozens of us!

> There are literally dozens of us!

I am happy that you are here but there is no need for hyperbole.

Everyone except most developers I know.

This is oddly true in my experience. I ask people where they get their information from and the channels are generally a lot more sparse than topics posted to HN.

If I were to venture to guess: I'd say "most developers" seek work-life balance and aren't interested in reading long-form articles on how to do X or Y when they are off the clock.

I worked with COBOL in banking, a lifetime ago. It was one of my first jobs.

Batch jobs, clunky and very verbose programs, nothing interesting. I... hated it.

Cloud migrations

Rewriting things in Java? Or maybe running Cobol in the cloud directly? A mainframe has a number of properties of the cloud, in a sense.

I knew a guy who wrote a lot of 360 assembler back in the day, never a COBOL programmer.

I was a consultant at a very large insurance company in the 90s when the dictate came from the top “no more assembler”. There were a few groans from the audience of developers.

Bank.

Can LLMs do Cobol?

Yep. I have recently used prompts to ask for COBOL code solution so I can compare it to other languages I also know to check quality of the answer. So far no mistakes.

> Can LLMs do Cobol?

I imagine it's the one place where LLMs would absolutely shine. COBOL jobs are usually very verbose, lots of boilerplate, but what they do is mostly very straightforward batch processing. It's ripe for automation with LLMs.

The flip side is that banks are usually very conservative about technology (for good reason).

I don't think LLMs are anything human language specific, so would they really shine here? I.e, COBOL and SQL may be great for humans who otherwise aren't used to programming languages, but LLMs have seen everything, and thus are able to know any (programming) language, not just ones which are English-like.

IMO the ideal path to working on COBOL without having decades of experience would be to spend a few days groking the syntax and writing toy programs for practice, then collaborate with a large LLM to understand the current code and gradually make changes.

Have you worked with COBOL? My understanding is that the language itself isn't the real problem. Mainframes, control languages: everything around COBOL is very different from most Windows/UNIX people have experienced.

Yes! But only if they write the compiler too.

not cobol but we do a hell of a lot of business basic.