Using deep learning to predict stock prices gains maturity

by Hansraj Agrawal | Aug 5, 2024 | Science

Alchemists of yore sought to short-cut their way to richness by finding ways to convert metals into gold. Today, it appears that the job of finding a quick pathway to prosperity has been taken up by data scientists. Alchemists, of course, failed to transmute cheap metals into gold, but data scientists may be sniffing success.

Today, it is possible to take data sets of quarterly results of a company, daily high, low and close prices and traded volumes, as well as economic data, news and sentiment analysis, and use the data to train a machine learning model to make predictions. The model essentially sees patterns that we would miss.

Data scientists first clean up the data (filling in any missing values and removing outliers) and make it usable by a model. Then they select a deep learning technique (like Long Short-Term Memory) or Convolutional Neural Networks, for milking the data for insights. LSTM is particularly appropriate for time series data, such as historical stock prices, because it can remember previous information and use it to make predictions. Then the model is trained. The training process involves making the model adjust its parameters to reduce the gap between its own predictions and actual stock prices. Then comes validation of the model, using a separate set of data to verify the model’s accuracy.

In broad terms, AI and deep learning predict stock prices by analysing historical data, identifying patterns and making future price forecasts. This involves collecting and pre-processing data, selecting and training appropriate models, making predictions, and continuously evaluating and refining the models to improve their accuracy and reliability.

Hitting the marks

In the last few years, a lot of work has been done in this area, with each model getting better than the earlier ones. A recent work in this area by three data scientists, Jaydip Sen, Hetvi Waghela and Sneha Rakshit, of the Department of Data Science and Artificial Intelligence at the Praxis Business School, Kolkata, has thrown up a model that boasts of 95.8 per cent accuracy in predicting the next day’s stock prices.

Sen, Waghela and Sneha used Long Short-term Memory (LSTM), which is a special kind of artificial neural network used in deep learning, designed to remember information for long periods of time, for “accurate stock price prediction”. In a recent paper (which is yet to be peer-reviewed), they note that “despite the efficient market hypothesis suggesting that such predictions are impossible, there are propositions in the literature demonstrating that advanced algorithms and predictive models can effectively forecast future stock prices.” They add that in recent times, the use of machine learning and deep learning systems has become popular for market price prediction.

Sen, Waghela and Rakshit believe that an LSTM model could predict stock prices better than other techniques such as convolutional neural networks that has been used by others earlier. “This model automatically retrieves historical stock prices using a Python function, utilising stock ticker names from the NSE within a specified interval determined by a start and end date.”

They took historical prices of 180 stocks across 18 sectors, between January 1, 2005 and April 23, 2024. They trained the data on LSTM model to make predictions. They used three metrics — Huber Loss, Mean Absolute Error and Accuracy Score — which give an idea of how wrong a prediction could be, to assess the performance of their model. For a model to be accurate, it must have low values for Huber loss and MAE, and a high value for the accuracy score.

Golden insight

Sen Waghela and Rakshit observed that their LSTM model yielded the minimum value of Huber loss for the auto sector, the minimum value of MAE for the banking sector, and the maximum value of accuracy score for the PSU banks sector. “Hence, the model is found to be the most accurate for these three sectors,” they note. On the other hand, Huber loss and MAE are the maximum for the media sector and accuracy score is the minimum for the energy sector. Therefore, the performance of the model has been the worst for media and energy sectors on the three metrics.

However, overall, the model’s performance has been highly accurate, since the lowest value of the accuracy score is 0.958315. “In other words, in the worst case, the model correctly predicted the direction of movement of the price (upward or downward) of the stock the next day in 95.83 per cent of cases. Thus, the model can be reliably used in stock trading decisions,” they say.

The use of deep learning techniques for making stock predictions will call for radical changes in regulations, so that the market does not get skewed in favour of those who know how to use techniques at the expense of those who don’t.

Source link

Non-toxic ionic liquids for silk processing

by Hansraj Agrawal | Jul 21, 2024 | Science

Researchers have found an eco-friendly approach that can eliminate the use of toxic chemicals in silk processing.

Traditionally, toxic chemicals like sodium carbonate, sodium hydroxide, sulfuric acid and lithium bromide have been used to extract silk proteins, fibroin and sericin from various types of raw silk fibres, an important step in the process of making silk from cocoons.

A team at the Institute of Advanced Study in Science and Technology (IASST), Guwahati, has identified Ionic Liquids (ILs) which can be sustainable alternatives to the toxic chemicals currently in use for the silk protein extraction process, according to a press release from the Department of Science and Technology, Government of India. The team, led by Dr Kamatchi Sankaranarayan, has identified four such ILs. Published in Chemistry Select, this research has potential for use in sericin extraction from both mulberry (Bombyx mori) and non-mulberry silks, such as Muga (Antheraea assamensis) and Eri (Philosamia ricini), indigenous to North-east India. Not only does it offer environmentally friendly alternative to traditional chemical methods, it also paves the way for efficient sericin extraction from non-mulberry silks, potentially leading to new applications for these unique fibres.

The researchers explored six different ILs and found some of them were particularly effective in removing sericin without damaging the silk protein structure. The ones showing greatest promise included 1-Butyl-3-methylimidazolium chloride (BMIM.Cl), 1-ethyl-3-methylimidazolium tetrafluoroborate (EMIM.BF4) and Tetraethylammonium bromide (TEAB). TEAB appeared to be highly effective due to its ability to destabilise sericin proteins.

Tackling battery power fade problem with Omics

Researchers at the Lawrence-Berkeley National Lab, California, US, have discovered a way to tackle the ‘power fade’ problem in batteries. The “power fade problem” in batteries refers to the gradual decrease in the battery’s ability to deliver power over time. The degradation affects the performance and efficiency of batteries, particularly in applications requiring high power output — which is why you have to throw away batteries after about five years. The power fade occurs due to factors such as ageing of electrodes and decomposition of electrolytes.

The L-B scientists took to ‘omics’ to study the power fade problem. The ‘Omics technique’ refers to a suite of technologies used to explore the roles, relationships, and actions of the various types of molecules that make up the cells of an organism.

The scientists “wanted to see if we could use a similar approach to examine the chemical signatures of the battery’s components and identify the reactions contributing to power fade and where they were occurring.”

The researchers focused their analysis on lithium metal batteries with high-voltage, high-density layered oxides containing nickel, manganese and cobalt. Contrary to prior research, which has typically thought the power fade problem was from the battery’s anode, the team observed that power fade stems from the cathode side. This was where particles cracked and corroded over time, hindering charge movement and reducing battery efficiency. “It was a non-obvious outcome,” Youngmin Ko, a postdoc researcher, said in a press release “We found that mixing salts in the electrolyte could suppress the reactivity of typically reactive species, which formed a stabilising, corrosion-resistant coating.”

Source link

Forward into past

by Hansraj Agrawal | Jul 21, 2024 | Science

You hail a taxi. A stylish cab called ‘Electrobat’ comes along. It is an electric vehicle.

A distance ahead, the driver stops, begs your forgiveness, for he needs just a jiffy to swap batteries. Three minutes later you are on your way, in a clean, green, electric vehicle — that has just avoided a huge pile of waste, the disposal of which has been a headache for the authorities.

Did I just take you to the future? Not at all. Quite the opposite — I took you to the past, to the year 1897, to New York’s infamous Manhattan, when electric vehicles, which we so covet these days, was the norm. Three years earlier, two engineers named Henry Morris and Pedro Salom, made an electric car and called it Electrobat. Rather than sell the vehicles, they got into running a taxi service. They called it — Electric Wagon & Carriage Company. The venture drove off with a dozen vehicles, but in just two years, was operating a hundred.

The battery it used was the old familiar lead-acid battery. It weighed a thousand pounds (454 kg, or about half as heavy as a Maruti Alto). With that weight, you would think that the vehicle can’t travel fast, but ‘fast’ is a relative term, conveying a different sense in every different context. In the context of 1899, it was an atrociously high speed of 25 km per hour. Guess what was the ‘waste’ that the vehicles avoided? Horse manure.

Electric Wagon & Carriage Co ran successfully until it lost its way due to a fraud, notes an article in National Geographic. In due course, internal combustion engines came and steamrolled the EVs out of business. But now, the EVs are having their sweet revenge.

Source link

De-identifying patients with AI: Ensuring patient privacy while enhancing research

by Hansraj Agrawal | Jul 21, 2024 | Science

In India, the most populous country in the world, efficient management — access, storage and retrieval — of healthcare data is increasingly critical. Imagine having access to health records of millions of patients, a treasure trove of information that could dramatically improve public health policies, advance medical research and enhance patient care. However, this also brings a significant challenge: protecting patient’s privacy.

A recent study “Generation and De-Identification of Indian Clinical Discharge Summaries using LLMs” by Sanjeet Singh et al, from the Indian Institute of Technology, Kanpur, (IIT Kanpur) and technology company Miimansa dives into this pressing issue.

The researchers explored how artificial intelligence (AI) can be harnessed to de-identify patient records, ensuring that sensitive information remains confidential while still being useful for research and policy-making.

Healthcare data is incredibly valuable. It can reveal patterns about the spread of diseases, the effectiveness of treatments, and the needs of different patient groups. In India, over 330 million patient records have already been linked with unique central IDs. This vast amount of data, roughly equivalent to the population of the US, represents an underutilised resource with the potential to revolutionise public health. However, it also poses a risk. If not handled properly, this data can expose individuals to privacy breaches. The consequences can be severe, from personal embarrassment to identity theft and financial loss.

To mitigate these risks, healthcare data must be de-identified, stripping it of any personal information that could reveal the patient’s identity. Natural Language Processing (NLP), a branch of AI that deals with the interaction between computers and human language, offers powerful tools for de-identification. NLP can scan through text, identify personal health information (PHI), and mask it.

However, there’s a catch: AI systems are only as good as the data they are trained on. Most existing systems have been trained on data from Western countries and they might not perform well on Indian data, given the cultural and linguistic differences.

De-identification of personal health information (PHI) is also critical to ensure compliance with privacy regulations such as the Indian Digital Personal Data Protection Act, 2023, (DPDPA) and similar laws like GDPR in Europe and HIPAA in the US.

The study from IIT Kanpur and Miimansa tackled this challenge head-on. Using a dataset of fully de-identified discharge summaries from an Indian hospital (the Sanjay Gandhi Post Graduate Institute of Medical Sciences, Lucknow), the researchers ran existing de-identification models, including commercial solutions. These models were originally trained on non-Indian datasets which primarily included data from US healthcare institutions. The results were telling: Models trained on non-Indian data did not perform well — a clear indication that AI models need to be trained on region-specific data to be effective.

Synthetic solution

To overcome this limitation, the researchers turned to a clever solution: synthetic data. By using large language models (LLMs) like Gemini, Gemma, Mistral and Llama3, they generated synthetic clinical reports that mimicked real patient data but did not correspond to actual patients, avoiding privacy issues. Training AI models on synthetic data dramatically improved their performance on the real Indian data.

This approach also ensures that healthcare data can be used safely for research and policy-making without risking patient privacy. For India, this could mean more accurate health statistics and better public health interventions.

While the results of this study are promising, there is still a long way to go. AI systems need continuous improvement and validation. The researchers plan to establish an active learning workflow that combines AI models with human expertise. This means that while AI will do the heavy lifting, human experts will refine and validate the results, creating a feedback loop that continuously enhances the system’s accuracy and reliability.

In a country as diverse and populous as India, blend of technology and human touch will be crucial in building a robust, resilient and responsive healthcare system.

Source link

Revolutionising TB research with 3D hydrogel model of lungs

by Hansraj Agrawal | Jul 21, 2024 | Science

“It is a very old bug and has evolved with us quite a bit,” says Rachit Agarwal, Associate Professor at the Department of Bioengineering, Indian Institute of Science, referring to Mycobacterium tuberculosis, a pathogen that kills 1.3 million every year.

Microbes mutate and develop resistance to existing drugs. When you develop new drugs, you need to test them.

Clinicians culture the bacteria on glass plates or petri dishes, but these do not properly mimic the 3D microenvironment inside lungs. Vishal Gupta, a PhD student explains the situation thus: “In a tissue culture plate, there are no extracellular matrix (ECM) molecules, and even if a very thin layer of ECM is coated on these plates, the lung cells ‘see’ the ECM on one side at best.”

To give those who culture the bacteria an environment closer to reality, researchers from the Department of Bioengineering, IISc, Bengaluru, have designed a 3D hydrogel culture system that “mimics the mammalian lung environment,” says a press release.

Jelly genius

Hydrogel is a material produced by carefully removing the liquids from a jelly, leaving a porous structure that has advantages such as resistance to extreme heat.

The 3D hydrogel developed by IISc scientists is made of collagen, a key molecule present in the ECM of lung cells. Collagen is soluble in water at a slightly acidic pH. As the pH increases, the collagen forms fibrils which cross-link to form a gel-like 3D structure. At the time of gelling, the researchers added human macrophages (immune cells involved in fighting infection) along with the tuberculosis causing pathogen. This entrapped both the macrophages and the bacteria in the collagen allowing researchers to track how the microbe infects the macrophages, the IISc write-up says. Using this set up, the team tracked the progress of an infection over 2-3 weeks. They found that the mammalian cells stayed viable for three weeks. Contrast this with just about a week when they did it by conventional culture methods. Further, the researchers carried out RNS sequencing of the lung cells that grew in hydrogel. They found that the cells were very similar to actual human samples.

The team also tested the effect of pyrazinamide — one of the four most common drugs given to TB patients. They found that even a small amount (10 µg/ml) of the drug was quite effective in clearing out the TB pathogen in the hydrogel culture, says IISc.

Previously, scientists have had to use large doses of the drug — much higher compared to concentrations achieved in patients — to show that it is effective in tissue culture. “Nobody has shown that this drug works in clinically relevant doses in any culture systems… Our setup reinforces the fact that the 3D hydrogel mimics the infection better,” observes Rachit Agarwal.

Source link

Conquering the skies with flying taxis

by Hansraj Agrawal | Jul 14, 2024 | Science

It’s a common sight to see small size drones with a carrying capacity of 5 to 10 kg in the sky spraying pesticides, or involved in surveillance of critical infrastructure or delivering drugs in to remote areas. But, the IIT Madras-incubated ePlane is trying to disrupt the sector with its eVTOLs (electric Vertical Take-off and Landing) — an electric drone that can carry both cargo and passengers. The 3X3 m sized muti-copter drone can carry 35 to 50 kg of cargo, travel at 400 ft and up to 50 km.

ePlane, which got a funding of $5-million for the project, is building India’s first and the world’s most compact flying electric taxi with a vision to make flying ubiquitous, says Satyanarayanan R Chakravarthy, Professor, Department of Aerospace Engineering, IIT-Madras, and Founder Director of ePlane.

If a car takes an hour to reach a destination, or a helicopter 27 minutes, eVTOL will take just 14 minutes, claims the ePlane team.

eVTOL will serve short haul mobility and urban mobility where there is a traffic congestion problem, says Prof Satyanarayanan, adding that electric aviation will disrupt the aviation sector in the foreseeable future.

From India, for the world

“What is good for India is good for the world. India should do it ahead of others. The country has UPI; the Aadhar stack and EVMs, then why not electric aviation,” he asks.

The five-year-old deep-tech start-up has developed a subscale prototype which it demonstrated last year. It is now working on a commercial version. “We are on the verge of flying that in the next few weeks, and then we will commercialise it. The subscale version is not meant for passenger travel but for cargo. We will tap the logistics players to adopt it. We have to go through a certification process for that as well,” he says.

“We are working on the passenger version, getting into the detailed design phase now. We will get into prototyping later this year. By early next year, we should have the first passenger prototype,” says Prof Satyanarayanan.

The company has built autonomous flight paths for collision avoidance. It will also set up autonomous Air Traffic Control with manual override for safe landing at various locations, he noted.

The subscale prototype will be governed by the drone rules. Although it can go at high altitudes of around 5,000 ft, it can be flown under 400 ft as well. There may not be much hindrance in India as most of the buildings are 50 m to 100 m tall, he explained.

The 50 kg payload could be the mid-mile segment for clients like logistics players and parcel delivering companies. “We are not going to replace all of the cargo movement. We feel that precious cargo, time critical cargo, medical supplies and organs between hospitals are a few examples where we can come in,” he said.

Spreading wings

Drone is a competitive space with different players deploying drones of different sizes and varieties. However, eVTOL flies with wings to cover longer distances. The key is to test the commercial prototype. “We are a few weeks away from its flight test of at least 100 hours. Then we will go through the certification process. We are a few months away from commercialisation,” he added.

Source link

« Older Entries

Next Entries »

Times Top News

Times Top News