Buy Crypto
Pay with
USD
USD
Buy & Sell
Hot
Supports Visa, Mastercard, SEPA & more
P2P
0 Fees
Flexible trading, zero fees
Gate Card
Use your crypto for payments worldwide
Markets
Trade
Basic
Advanced
Futures
Futures
Access hundreds of perpetual contracts
CFD
Gold
One platform for global traditional assets
Options
Hot
Trade European-style vanilla options
Unified Account
Maximize your capital efficiency
Demo Trading
Introduction to Futures Trading
Learn the basics of futures trading
Futures Events
Join events to earn rewards
Demo Trading
Use virtual funds to practice risk-free trading
Stocks
CFD
Stock CFD Derivatives
US Stocks
Access real US stocks and ETFs
HK Stocks
Trade quality Hong Kong-listed stocks
Korean Stocks
SK Hynix
Real Korean stocks and top assets
Stock Futures
High leverage, 24/7 trading
Tokenized Stocks
Backed by real stock assets
IPO Access
Unlock full access to global stock IPOs
GUSD
3.8%
Mint GUSD for Treasury RWA yields
Stocks Activities
Trade Popular Stocks and Unlock Generous Airdrops
Earn
Launch
CandyDrop
tag
Collect candies to earn airdrops
Launchpool
Quick staking, earn potential new tokens
HODLer Airdrop
Hold GT and get massive airdrops for free
IPO Access
Unlock full access to global stock IPOs
Alpha Points
Trade on-chain assets and earn airdrops
Futures Points
Earn futures points and claim airdrop rewards
Investment
Square
Square
Post, share, and explore crypto trends
Live
moments live
Live crypto market analysis
Chat
Chat with crypto traders
News
What is happening in crypto
flower_head
More
Promotions
AI
Others
TradFi
2026 World Cup
Rewards

OpenAI Finds About 30% of SWE-Bench Pro Questions Flawed, Withdraws Recommendation

GateNews
AI Industry News
According to an audit report released by OpenAI, approximately 30% of the 731 public questions in SWE-Bench Pro are unsuitable for reliable AI programming evaluation. Automated screening identified 200 problematic questions (27.4%), while manual review found 249 issues (34.1%). Main problems include incomplete task descriptions, overly strict test requirements, hidden tests checking undisclosed requirements, and insufficient test coverage. OpenAI noted that cutting-edge models' pass rate on these public questions rose from 23.3% to 80.3% over eight months, but such gains warrant cautious interpretation given the benchmark's noise. The company has withdrawn its previous recommendation to adopt SWE-Bench Pro and called for the community to build more reliable programming benchmarks.
View Source
Disclaimer: The information on this page may come from third-party sources and is for reference only. It does not represent the views or opinions of Gate and does not constitute any financial, investment, or legal advice. Virtual asset trading involves high risk. Please do not rely solely on the information on this page when making decisions. For details, see the Disclaimer.

Related News

1h ago
Cognition Releases SWE-1.7 Programming Model Reaching 1000 Tokens Per Second
10h ago
Google Updates Android Bench with Eight New Models; Claude Fable 5 Tops Leaderboard at 84.5% Accuracy
18h ago
U.S. Semiconductor Industry Faces 157,000-Worker Shortage by 2030
19h ago
Singapore's AI Daily Usage at Only 6%, Below Global Average of 11%
20h ago
Chinese AI Models Priced 60%-90% Lower Than U.S. Competitors, Surging Adoption Among American Enterprises
In-Depth Analysis

SpaceXAI releases Grok 4.5: jointly trained with Cursor, integrated with Microsoft Office

Market Whisper1h ago

ILO Finds 80 Million ASEAN Workers Face GenAI Exposure in New Policy Brief

Oliver Grant19h ago

Morgan Stanley: Power Grid Access Becomes AI Industry's Next Stress Test

Oliver Grant07-08 01:03
Comment
0/400
No comments